[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790194121577027-1706-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 20:08:41 dist-test-slave-th4g krb5kdc[1713](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 20:08:41 dist-test-slave-th4g krb5kdc[1713](info): set up 2 sockets
Sep 23 20:08:41 dist-test-slave-th4g krb5kdc[1713](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260923 20:08:42.762363 1706 mini_kdc.cc:121] Time spent starting KDC: real 1.064s user 0.005s sys 0.014s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790194121577027-1706-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790194121577027-1706-0/krb5kdc/test-user.keytab.
Sep 23 20:08:42 dist-test-slave-th4g krb5kdc[1713](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194122, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T20:08:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T20:08:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.170.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.170.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.170.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790194121577027-1706-0/krb5kdc/kudu_127.1.170.190.keytab.
Entry for principal kudu/127.1.170.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790194121577027-1706-0/krb5kdc/kudu_127.1.170.190.keytab.
I20260923 20:08:42.946209 1706 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqUuku6/build/asan/bin/kudu
/tmp/dist-test-taskqUuku6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790194121577027-1706-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790194121577027-1706-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790194121577027-1706-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790194121577027-1706-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.170.190:40663
--webserver_interface=127.1.170.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790194121577027-1706-0/krb5kdc/kudu_127.1.170.190.keytab
--principal=kudu/127.1.170.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.170.148:44881
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.170.190:40663
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790194121577027-1706-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790194121577027-1706-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790194121577027-1706-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:08:43.297464 1726 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:08:43.297797 1726 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:08:43.298105 1726 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:08:43.305627 1726 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 20:08:43.305743 1726 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:08:43.305868 1726 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:08:43.305979 1726 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 20:08:43.306066 1726 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 20:08:43.316637 1726 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.170.148:44881
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790194121577027-1706-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790194121577027-1706-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.170.190:40663
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790194121577027-1706-0/krb5kdc/kudu_127.1.170.190.keytab
--principal=kudu/127.1.170.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.170.190:40663
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790194121577027-1706-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.170.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790194121577027-1706-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:08:43.318487 1726 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:08:43.320164 1726 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:08:43.343902 1734 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:08:43.343853 1731 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:08:43.343801 1732 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:08:43.346108 1726 server_base.cc:1034] running on GCE node
I20260923 20:08:43.347569 1726 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:08:43.349444 1726 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:08:43.350692 1726 hybrid_clock.cc:648] HybridClock initialized: now 1790194123350611 us; error 69 us; skew 500 ppm
I20260923 20:08:43.351220 1726 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:08:43 dist-test-slave-th4g krb5kdc[1713](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194123, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.170.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:08:43.354571 1726 init.cc:377] Logged in from keytab as kudu/127.1.170.190@KRBTEST.COM (short username kudu)
I20260923 20:08:43.356704 1726 webserver.cc:460] Webserver started at http://127.1.170.190:38827/ using document root <none> and password file <none>
I20260923 20:08:43.357575 1726 fs_manager.cc:359] Metadata directory not provided
I20260923 20:08:43.357728 1726 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:08:43.358156 1726 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:08:43.362077 1726 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790194121577027-1706-0/minicluster-data/master-0/data/instance:
uuid: "8429d90c5a314b8fb43dfd964fc51654"
format_stamp: "Formatted at 2026-09-23 20:08:43 on dist-test-slave-th4g"
server_key: "3a5472e5c44f09b7858c34a5327b14c1"
server_key_iv: "f9d6019bdbb13e885e4730f48c48758e"
server_key_version: "encryptionkey@0"
I20260923 20:08:43.363067 1726 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790194121577027-1706-0/minicluster-data/master-0/wal/instance:
uuid: "8429d90c5a314b8fb43dfd964fc51654"
format_stamp: "Formatted at 2026-09-23 20:08:43 on dist-test-slave-th4g"
server_key: "3a5472e5c44f09b7858c34a5327b14c1"
server_key_iv: "f9d6019bdbb13e885e4730f48c48758e"
server_key_version: "encryptionkey@0"
I20260923 20:08:43.369982 1726 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.001s
I20260923 20:08:43.374994 1741 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:08:43.376982 1726 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.002s
I20260923 20:08:43.377358 1726 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790194121577027-1706-0/minicluster-data/master-0/data,/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790194121577027-1706-0/minicluster-data/master-0/wal
uuid: "8429d90c5a314b8fb43dfd964fc51654"
format_stamp: "Formatted at 2026-09-23 20:08:43 on dist-test-slave-th4g"
server_key: "3a5472e5c44f09b7858c34a5327b14c1"
server_key_iv: "f9d6019bdbb13e885e4730f48c48758e"
server_key_version: "encryptionkey@0"
I20260923 20:08:43.377589 1726 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790194121577027-1706-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790194121577027-1706-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790194121577027-1706-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:08:43.392889 1726 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:08:43.397179 1726 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:08:43.397519 1726 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:08:43.419236 1726 rpc_server.cc:307] RPC server started. Bound to: 127.1.170.190:40663
I20260923 20:08:43.419242 1792 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.170.190:40663 every 8 connection(s)
I20260923 20:08:43.421204 1726 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790194121577027-1706-0/minicluster-data/master-0/data/info.pb
I20260923 20:08:43.423954 1706 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqUuku6/build/asan/bin/kudu as pid 1726
I20260923 20:08:43.424279 1706 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790194121577027-1706-0/minicluster-data/master-0/wal/instance
I20260923 20:08:43.426496 1706 external_mini_cluster.cc:1428] Setting key 107e58cfee65239dafa61e8f18513eeb
I20260923 20:08:43.427079 1793 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:08:43.443907 1793 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8429d90c5a314b8fb43dfd964fc51654: Bootstrap starting.
I20260923 20:08:43.449216 1793 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8429d90c5a314b8fb43dfd964fc51654: Neither blocks nor log segments found. Creating new log.
I20260923 20:08:43.450870 1793 log.cc:824] T 00000000000000000000000000000000 P 8429d90c5a314b8fb43dfd964fc51654: Log is configured to *not* fsync() on all Append() calls
I20260923 20:08:43.454387 1793 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8429d90c5a314b8fb43dfd964fc51654: No bootstrap required, opened a new log
Sep 23 20:08:43 dist-test-slave-th4g krb5kdc[1713](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194122, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.170.190@KRBTEST.COM
I20260923 20:08:43.465106 1793 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8429d90c5a314b8fb43dfd964fc51654 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8429d90c5a314b8fb43dfd964fc51654" member_type: VOTER last_known_addr { host: "127.1.170.190" port: 40663 } }
I20260923 20:08:43.465566 1793 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8429d90c5a314b8fb43dfd964fc51654 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:08:43.465745 1793 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8429d90c5a314b8fb43dfd964fc51654 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8429d90c5a314b8fb43dfd964fc51654, State: Initialized, Role: FOLLOWER
I20260923 20:08:43.466863 1793 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8429d90c5a314b8fb43dfd964fc51654 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8429d90c5a314b8fb43dfd964fc51654" member_type: VOTER last_known_addr { host: "127.1.170.190" port: 40663 } }
I20260923 20:08:43.467190 1793 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8429d90c5a314b8fb43dfd964fc51654 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 20:08:43.467355 1793 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8429d90c5a314b8fb43dfd964fc51654 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 20:08:43.467538 1793 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8429d90c5a314b8fb43dfd964fc51654 [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:08:43.470005 1793 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8429d90c5a314b8fb43dfd964fc51654 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8429d90c5a314b8fb43dfd964fc51654" member_type: VOTER last_known_addr { host: "127.1.170.190" port: 40663 } }
I20260923 20:08:43.470820 1793 leader_election.cc:302] T 00000000000000000000000000000000 P 8429d90c5a314b8fb43dfd964fc51654 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8429d90c5a314b8fb43dfd964fc51654; no voters:
I20260923 20:08:43.471692 1793 leader_election.cc:288] T 00000000000000000000000000000000 P 8429d90c5a314b8fb43dfd964fc51654 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 20:08:43.471743 1795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:08:43.431630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60688 (local address 127.1.170.190:40663)
0923 20:08:43.433262 (+ 1632us) server_negotiation.cc:206] Beginning negotiation
0923 20:08:43.433277 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0923 20:08:43.434338 (+ 1061us) server_negotiation.cc:407] Connection header received
0923 20:08:43.446148 (+ 11810us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:08:43.446225 (+ 77us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:08:43.446708 (+ 483us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:08:43.447205 (+ 497us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:08:43.450386 (+ 3181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:08:43.452002 (+ 1616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:08:43.453767 (+ 1765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:08:43.454489 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:08:43.454695 (+ 206us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:08:43.462573 (+ 7878us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:08:43.462583 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:08:43.462598 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:08:43.462648 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:08:43.466024 (+ 3376us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:08:43.466856 (+ 832us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:08:43.466865 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:08:43.466873 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:08:43.467060 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:08:43.467763 (+ 703us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:08:43.467768 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:08:43.467772 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:08:43.468475 (+ 703us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:08:43.468838 (+ 363us) server_negotiation.cc:1035] Waiting for connection context
0923 20:08:43.469772 (+ 934us) server_negotiation.cc:299] Negotiation successful
0923 20:08:43.470196 (+ 424us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1243,"thread_start_us":375,"threads_started":1}
I20260923 20:08:43.472024 1798 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8429d90c5a314b8fb43dfd964fc51654 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 20:08:43.474268 1798 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8429d90c5a314b8fb43dfd964fc51654 [term 1 LEADER]: Becoming Leader. State: Replica: 8429d90c5a314b8fb43dfd964fc51654, State: Running, Role: LEADER
I20260923 20:08:43.474969 1798 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8429d90c5a314b8fb43dfd964fc51654 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8429d90c5a314b8fb43dfd964fc51654" member_type: VOTER last_known_addr { host: "127.1.170.190" port: 40663 } }
I20260923 20:08:43.477990 1798 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8429d90c5a314b8fb43dfd964fc51654 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8429d90c5a314b8fb43dfd964fc51654. Latest consensus state: current_term: 1 leader_uuid: "8429d90c5a314b8fb43dfd964fc51654" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8429d90c5a314b8fb43dfd964fc51654" member_type: VOTER last_known_addr { host: "127.1.170.190" port: 40663 } } }
I20260923 20:08:43.478364 1798 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8429d90c5a314b8fb43dfd964fc51654 [sys.catalog]: This master's current role is: LEADER
I20260923 20:08:43.479667 1800 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8429d90c5a314b8fb43dfd964fc51654 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8429d90c5a314b8fb43dfd964fc51654" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8429d90c5a314b8fb43dfd964fc51654" member_type: VOTER last_known_addr { host: "127.1.170.190" port: 40663 } } }
I20260923 20:08:43.480036 1800 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8429d90c5a314b8fb43dfd964fc51654 [sys.catalog]: This master's current role is: LEADER
I20260923 20:08:43.482254 1793 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8429d90c5a314b8fb43dfd964fc51654 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 20:08:43.482761 1802 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 20:08:43.490490 1802 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 20:08:43.506477 1802 catalog_manager.cc:1347] Generated new cluster ID: 647a4c0c29b541ec9eb57a1a69f41c17
I20260923 20:08:43.506613 1802 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 20:08:43.524802 1802 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 20:08:43.526258 1802 catalog_manager.cc:1504] Loading token signing keys...
I20260923 20:08:43.538075 1802 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8429d90c5a314b8fb43dfd964fc51654: Generated new TSK 0
I20260923 20:08:43.539251 1802 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.170.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.170.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.170.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790194121577027-1706-0/krb5kdc/kudu_127.1.170.129.keytab.
Entry for principal kudu/127.1.170.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790194121577027-1706-0/krb5kdc/kudu_127.1.170.129.keytab.
I20260923 20:08:43.590951 1706 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqUuku6/build/asan/bin/kudu
/tmp/dist-test-taskqUuku6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790194121577027-1706-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790194121577027-1706-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790194121577027-1706-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790194121577027-1706-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.170.129:0
--local_ip_for_outbound_sockets=127.1.170.129
--webserver_interface=127.1.170.129
--webserver_port=0
--tserver_master_addrs=127.1.170.190:40663
--keytab_file=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790194121577027-1706-0/krb5kdc/kudu_127.1.170.129.keytab
--principal=kudu/127.1.170.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.170.148:44881
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790194121577027-1706-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790194121577027-1706-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790194121577027-1706-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:08:43.928543 1819 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:08:43.928817 1819 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:08:43.929386 1819 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:08:43.937323 1819 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:08:43.937435 1819 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:08:43.937609 1819 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.170.129
I20260923 20:08:43.947530 1819 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.170.148:44881
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790194121577027-1706-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790194121577027-1706-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790194121577027-1706-0/krb5kdc/kudu_127.1.170.129.keytab
--principal=kudu/127.1.170.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.170.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790194121577027-1706-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.170.129
--webserver_port=0
--tserver_master_addrs=127.1.170.190:40663
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.170.129
--log_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790194121577027-1706-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:08:43.948863 1819 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:08:43.950286 1819 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:08:43.964882 1824 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:08:43.964807 1827 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:08:43.964845 1825 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:08:43.965725 1819 server_base.cc:1034] running on GCE node
I20260923 20:08:43.966387 1819 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:08:43.967386 1819 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:08:43.968626 1819 hybrid_clock.cc:648] HybridClock initialized: now 1790194123968575 us; error 54 us; skew 500 ppm
I20260923 20:08:43.968950 1819 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:08:43 dist-test-slave-th4g krb5kdc[1713](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194123, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.170.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:08:43.972450 1819 init.cc:377] Logged in from keytab as kudu/127.1.170.129@KRBTEST.COM (short username kudu)
I20260923 20:08:43.974030 1819 webserver.cc:460] Webserver started at http://127.1.170.129:42905/ using document root <none> and password file <none>
I20260923 20:08:43.974757 1819 fs_manager.cc:359] Metadata directory not provided
I20260923 20:08:43.974905 1819 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:08:43.975371 1819 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:08:43.978435 1819 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790194121577027-1706-0/minicluster-data/ts-0/data/instance:
uuid: "abc1f60b7b364f5c896790297259483e"
format_stamp: "Formatted at 2026-09-23 20:08:43 on dist-test-slave-th4g"
server_key: "c34525219215741136975a856c193e07"
server_key_iv: "212496cf9feed12ac4a4e90e2575107e"
server_key_version: "encryptionkey@0"
I20260923 20:08:43.979236 1819 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790194121577027-1706-0/minicluster-data/ts-0/wal/instance:
uuid: "abc1f60b7b364f5c896790297259483e"
format_stamp: "Formatted at 2026-09-23 20:08:43 on dist-test-slave-th4g"
server_key: "c34525219215741136975a856c193e07"
server_key_iv: "212496cf9feed12ac4a4e90e2575107e"
server_key_version: "encryptionkey@0"
I20260923 20:08:43.984942 1819 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260923 20:08:43.988531 1834 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:08:43.990000 1819 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 20:08:43.990175 1819 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790194121577027-1706-0/minicluster-data/ts-0/data,/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790194121577027-1706-0/minicluster-data/ts-0/wal
uuid: "abc1f60b7b364f5c896790297259483e"
format_stamp: "Formatted at 2026-09-23 20:08:43 on dist-test-slave-th4g"
server_key: "c34525219215741136975a856c193e07"
server_key_iv: "212496cf9feed12ac4a4e90e2575107e"
server_key_version: "encryptionkey@0"
I20260923 20:08:43.990418 1819 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790194121577027-1706-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790194121577027-1706-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790194121577027-1706-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:08:44.004127 1819 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:08:44.010648 1819 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:08:44.010959 1819 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:08:44.012171 1819 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 20:08:44.013810 1819 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 20:08:44.013953 1819 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260923 20:08:44.014158 1819 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 20:08:44.014237 1819 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 20:08:44.050945 1819 rpc_server.cc:307] RPC server started. Bound to: 127.1.170.129:35167
I20260923 20:08:44.051215 1946 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.170.129:35167 every 8 connection(s)
I20260923 20:08:44.052280 1819 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790194121577027-1706-0/minicluster-data/ts-0/data/info.pb
I20260923 20:08:44.059247 1706 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqUuku6/build/asan/bin/kudu as pid 1819
I20260923 20:08:44.059561 1706 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790194121577027-1706-0/minicluster-data/ts-0/wal/instance
I20260923 20:08:44.060187 1706 external_mini_cluster.cc:1428] Setting key e96f0f0bb83f5e3b1cbd70af4633142d
Sep 23 20:08:44 dist-test-slave-th4g krb5kdc[1713](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194123, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.170.129@KRBTEST.COM for kudu/127.1.170.190@KRBTEST.COM
I20260923 20:08:44.076754 1949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:08:44.055752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.170.129:38609 (local address 127.1.170.190:40663)
0923 20:08:44.056220 (+ 468us) server_negotiation.cc:206] Beginning negotiation
0923 20:08:44.056225 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 20:08:44.057180 (+ 955us) server_negotiation.cc:407] Connection header received
0923 20:08:44.059480 (+ 2300us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:08:44.059486 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:08:44.059581 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:08:44.059703 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:08:44.061406 (+ 1703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:08:44.062217 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:08:44.063868 (+ 1651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:08:44.064305 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:08:44.064444 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:08:44.068788 (+ 4344us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:08:44.068798 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:08:44.068802 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:08:44.068848 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:08:44.072687 (+ 3839us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:08:44.073574 (+ 887us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:08:44.073583 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:08:44.073589 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:08:44.073742 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:08:44.074355 (+ 613us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:08:44.074363 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:08:44.074369 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:08:44.074796 (+ 427us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:08:44.074978 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0923 20:08:44.076024 (+ 1046us) server_negotiation.cc:299] Negotiation successful
0923 20:08:44.076307 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":341,"thread_start_us":230,"threads_started":1}
I20260923 20:08:44.077482 1950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:08:44.056301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.190:40663 (local address 127.1.170.129:38609)
0923 20:08:44.057012 (+ 711us) negotiation.cc:107] Waiting for socket to connect
0923 20:08:44.057054 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0923 20:08:44.059174 (+ 2120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:08:44.059949 (+ 775us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:08:44.059958 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:08:44.060542 (+ 584us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:08:44.061241 (+ 699us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:08:44.061262 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:08:44.062387 (+ 1125us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:08:44.062397 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:08:44.063669 (+ 1272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:08:44.063686 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:08:44.064536 (+ 850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:08:44.064546 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:08:44.064786 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:08:44.065492 (+ 706us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:08:44.065515 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:08:44.068513 (+ 2998us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:08:44.072867 (+ 4354us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:08:44.072875 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:08:44.072891 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:08:44.073439 (+ 548us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:08:44.073938 (+ 499us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:08:44.073943 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:08:44.074011 (+ 68us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:08:44.074206 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:08:44.074951 (+ 745us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:08:44.074957 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:08:44.075535 (+ 578us) client_negotiation.cc:769] Sending connection context
0923 20:08:44.075864 (+ 329us) client_negotiation.cc:240] Negotiation successful
0923 20:08:44.076289 (+ 425us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":495,"thread_start_us":285,"threads_started":1}
WARNING: no policy specified for kudu/127.1.170.130@KRBTEST.COM; defaulting to no policy
I20260923 20:08:44.079459 1947 heartbeater.cc:347] Connected to a master server at 127.1.170.190:40663
I20260923 20:08:44.079938 1947 heartbeater.cc:464] Registering TS with master...
I20260923 20:08:44.080880 1947 heartbeater.cc:511] Master 127.1.170.190:40663 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.170.130@KRBTEST.COM" created.
I20260923 20:08:44.085467 1758 ts_manager.cc:194] Registered new tserver with Master: abc1f60b7b364f5c896790297259483e (127.1.170.129:35167)
I20260923 20:08:44.087900 1758 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.170.129@KRBTEST.COM'} at 127.1.170.129:38609
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.170.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790194121577027-1706-0/krb5kdc/kudu_127.1.170.130.keytab.
Entry for principal kudu/127.1.170.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790194121577027-1706-0/krb5kdc/kudu_127.1.170.130.keytab.
I20260923 20:08:44.107016 1706 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqUuku6/build/asan/bin/kudu
/tmp/dist-test-taskqUuku6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790194121577027-1706-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790194121577027-1706-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790194121577027-1706-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790194121577027-1706-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.170.130:0
--local_ip_for_outbound_sockets=127.1.170.130
--webserver_interface=127.1.170.130
--webserver_port=0
--tserver_master_addrs=127.1.170.190:40663
--keytab_file=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790194121577027-1706-0/krb5kdc/kudu_127.1.170.130.keytab
--principal=kudu/127.1.170.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.170.148:44881
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790194121577027-1706-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790194121577027-1706-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790194121577027-1706-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:08:44.451596 1953 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:08:44.451874 1953 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:08:44.451972 1953 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:08:44.459533 1953 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:08:44.459653 1953 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:08:44.459767 1953 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.170.130
I20260923 20:08:44.470146 1953 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.170.148:44881
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790194121577027-1706-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790194121577027-1706-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790194121577027-1706-0/krb5kdc/kudu_127.1.170.130.keytab
--principal=kudu/127.1.170.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.170.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790194121577027-1706-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.170.130
--webserver_port=0
--tserver_master_addrs=127.1.170.190:40663
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.170.130
--log_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790194121577027-1706-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:08:44.471418 1953 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:08:44.472659 1953 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:08:44.486615 1959 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:08:44.486565 1958 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:08:44.486565 1961 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:08:44.487922 1953 server_base.cc:1034] running on GCE node
I20260923 20:08:44.488546 1953 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:08:44.489507 1953 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:08:44.490741 1953 hybrid_clock.cc:648] HybridClock initialized: now 1790194124490691 us; error 43 us; skew 500 ppm
I20260923 20:08:44.491129 1953 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:08:44 dist-test-slave-th4g krb5kdc[1713](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194124, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.170.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:08:44.495121 1953 init.cc:377] Logged in from keytab as kudu/127.1.170.130@KRBTEST.COM (short username kudu)
I20260923 20:08:44.496956 1953 webserver.cc:460] Webserver started at http://127.1.170.130:42475/ using document root <none> and password file <none>
I20260923 20:08:44.497699 1953 fs_manager.cc:359] Metadata directory not provided
I20260923 20:08:44.497848 1953 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:08:44.498178 1953 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:08:44.501021 1953 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790194121577027-1706-0/minicluster-data/ts-1/data/instance:
uuid: "9d182fb36f524b08961a70e7e5d0a3e9"
format_stamp: "Formatted at 2026-09-23 20:08:44 on dist-test-slave-th4g"
server_key: "7ad52659a9a0230e700e6c6f2f1f863a"
server_key_iv: "285e07319e8ce5aaa7ebe6fe5738b051"
server_key_version: "encryptionkey@0"
I20260923 20:08:44.501775 1953 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790194121577027-1706-0/minicluster-data/ts-1/wal/instance:
uuid: "9d182fb36f524b08961a70e7e5d0a3e9"
format_stamp: "Formatted at 2026-09-23 20:08:44 on dist-test-slave-th4g"
server_key: "7ad52659a9a0230e700e6c6f2f1f863a"
server_key_iv: "285e07319e8ce5aaa7ebe6fe5738b051"
server_key_version: "encryptionkey@0"
I20260923 20:08:44.507136 1953 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.003s
I20260923 20:08:44.510643 1968 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:08:44.511933 1953 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260923 20:08:44.512092 1953 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790194121577027-1706-0/minicluster-data/ts-1/data,/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790194121577027-1706-0/minicluster-data/ts-1/wal
uuid: "9d182fb36f524b08961a70e7e5d0a3e9"
format_stamp: "Formatted at 2026-09-23 20:08:44 on dist-test-slave-th4g"
server_key: "7ad52659a9a0230e700e6c6f2f1f863a"
server_key_iv: "285e07319e8ce5aaa7ebe6fe5738b051"
server_key_version: "encryptionkey@0"
I20260923 20:08:44.512296 1953 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790194121577027-1706-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790194121577027-1706-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790194121577027-1706-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:08:44.533546 1953 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:08:44.538053 1953 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:08:44.538364 1953 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:08:44.539398 1953 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 20:08:44.540748 1953 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 20:08:44.540952 1953 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 20:08:44.541111 1953 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 20:08:44.541181 1953 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 20:08:44.578300 1953 rpc_server.cc:307] RPC server started. Bound to: 127.1.170.130:44139
I20260923 20:08:44.578351 2080 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.170.130:44139 every 8 connection(s)
I20260923 20:08:44.579614 1953 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790194121577027-1706-0/minicluster-data/ts-1/data/info.pb
I20260923 20:08:44.585722 1706 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqUuku6/build/asan/bin/kudu as pid 1953
I20260923 20:08:44.585963 1706 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790194121577027-1706-0/minicluster-data/ts-1/wal/instance
I20260923 20:08:44.586498 1706 external_mini_cluster.cc:1428] Setting key 50ff0c73838a09245a2446450535ac10
Sep 23 20:08:44 dist-test-slave-th4g krb5kdc[1713](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194124, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.170.130@KRBTEST.COM for kudu/127.1.170.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.1.170.131@KRBTEST.COM; defaulting to no policy
I20260923 20:08:44.605208 2083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:08:44.582479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.170.130:42791 (local address 127.1.170.190:40663)
0923 20:08:44.582947 (+ 468us) server_negotiation.cc:206] Beginning negotiation
0923 20:08:44.582951 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 20:08:44.584056 (+ 1105us) server_negotiation.cc:407] Connection header received
0923 20:08:44.586238 (+ 2182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:08:44.586243 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:08:44.586316 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:08:44.586440 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:08:44.588423 (+ 1983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:08:44.589164 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:08:44.592242 (+ 3078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:08:44.593404 (+ 1162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:08:44.593552 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:08:44.598815 (+ 5263us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:08:44.598821 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:08:44.598824 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:08:44.598860 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:08:44.601575 (+ 2715us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:08:44.602650 (+ 1075us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:08:44.602655 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:08:44.602657 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:08:44.602803 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:08:44.603343 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:08:44.603346 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:08:44.603348 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:08:44.603611 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:08:44.603729 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0923 20:08:44.604714 (+ 985us) server_negotiation.cc:299] Negotiation successful
0923 20:08:44.604895 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":361,"thread_start_us":244,"threads_started":1}
I20260923 20:08:44.606272 2084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:08:44.583083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.190:40663 (local address 127.1.170.130:42791)
0923 20:08:44.583876 (+ 793us) negotiation.cc:107] Waiting for socket to connect
0923 20:08:44.583921 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0923 20:08:44.585858 (+ 1937us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:08:44.586753 (+ 895us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:08:44.586765 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:08:44.587400 (+ 635us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:08:44.588192 (+ 792us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:08:44.588218 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:08:44.589329 (+ 1111us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:08:44.589335 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:08:44.592055 (+ 2720us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:08:44.592074 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:08:44.593688 (+ 1614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:08:44.593698 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:08:44.594086 (+ 388us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:08:44.594958 (+ 872us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:08:44.594997 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:08:44.598534 (+ 3537us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:08:44.601751 (+ 3217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:08:44.601764 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:08:44.601789 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:08:44.602481 (+ 692us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:08:44.602934 (+ 453us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:08:44.602940 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:08:44.602944 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:08:44.603210 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:08:44.603750 (+ 540us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:08:44.603760 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:08:44.604253 (+ 493us) client_negotiation.cc:769] Sending connection context
0923 20:08:44.604594 (+ 341us) client_negotiation.cc:240] Negotiation successful
0923 20:08:44.605028 (+ 434us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":524,"thread_start_us":332,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.170.131@KRBTEST.COM" created.
I20260923 20:08:44.607870 2081 heartbeater.cc:347] Connected to a master server at 127.1.170.190:40663
I20260923 20:08:44.608284 2081 heartbeater.cc:464] Registering TS with master...
I20260923 20:08:44.609054 2081 heartbeater.cc:511] Master 127.1.170.190:40663 requested a full tablet report, sending...
I20260923 20:08:44.610952 1758 ts_manager.cc:194] Registered new tserver with Master: 9d182fb36f524b08961a70e7e5d0a3e9 (127.1.170.130:44139)
I20260923 20:08:44.612160 1758 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.170.130@KRBTEST.COM'} at 127.1.170.130:42791
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.170.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790194121577027-1706-0/krb5kdc/kudu_127.1.170.131.keytab.
Entry for principal kudu/127.1.170.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790194121577027-1706-0/krb5kdc/kudu_127.1.170.131.keytab.
I20260923 20:08:44.634904 1706 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqUuku6/build/asan/bin/kudu
/tmp/dist-test-taskqUuku6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790194121577027-1706-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790194121577027-1706-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790194121577027-1706-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790194121577027-1706-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.170.131:0
--local_ip_for_outbound_sockets=127.1.170.131
--webserver_interface=127.1.170.131
--webserver_port=0
--tserver_master_addrs=127.1.170.190:40663
--keytab_file=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790194121577027-1706-0/krb5kdc/kudu_127.1.170.131.keytab
--principal=kudu/127.1.170.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.170.148:44881
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790194121577027-1706-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790194121577027-1706-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790194121577027-1706-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:08:45.006348 2087 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:08:45.006637 2087 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:08:45.006799 2087 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:08:45.014426 2087 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:08:45.014544 2087 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:08:45.014737 2087 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.170.131
I20260923 20:08:45.024796 2087 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.170.148:44881
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790194121577027-1706-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790194121577027-1706-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790194121577027-1706-0/krb5kdc/kudu_127.1.170.131.keytab
--principal=kudu/127.1.170.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.170.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790194121577027-1706-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.170.131
--webserver_port=0
--tserver_master_addrs=127.1.170.190:40663
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.170.131
--log_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790194121577027-1706-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:08:45.026165 2087 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:08:45.027611 2087 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:08:45.041222 2095 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:08:45.041297 2092 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:08:45.041209 2093 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:08:45.042655 2087 server_base.cc:1034] running on GCE node
I20260923 20:08:45.043386 2087 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:08:45.044374 2087 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:08:45.045701 2087 hybrid_clock.cc:648] HybridClock initialized: now 1790194125045643 us; error 47 us; skew 500 ppm
I20260923 20:08:45.046114 2087 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:08:45 dist-test-slave-th4g krb5kdc[1713](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194125, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.170.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:08:45.049777 2087 init.cc:377] Logged in from keytab as kudu/127.1.170.131@KRBTEST.COM (short username kudu)
I20260923 20:08:45.052181 2087 webserver.cc:460] Webserver started at http://127.1.170.131:43495/ using document root <none> and password file <none>
I20260923 20:08:45.053572 2087 fs_manager.cc:359] Metadata directory not provided
I20260923 20:08:45.053813 2087 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:08:45.054234 2087 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:08:45.057309 2087 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790194121577027-1706-0/minicluster-data/ts-2/data/instance:
uuid: "9c6dee09bc4242438716b3e97e15c0de"
format_stamp: "Formatted at 2026-09-23 20:08:45 on dist-test-slave-th4g"
server_key: "0c8c9e6041978c72621766006e52ee26"
server_key_iv: "99325d1c945118dbd40039ae313875cf"
server_key_version: "encryptionkey@0"
I20260923 20:08:45.058239 2087 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790194121577027-1706-0/minicluster-data/ts-2/wal/instance:
uuid: "9c6dee09bc4242438716b3e97e15c0de"
format_stamp: "Formatted at 2026-09-23 20:08:45 on dist-test-slave-th4g"
server_key: "0c8c9e6041978c72621766006e52ee26"
server_key_iv: "99325d1c945118dbd40039ae313875cf"
server_key_version: "encryptionkey@0"
I20260923 20:08:45.063791 2087 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260923 20:08:45.067627 2102 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:08:45.069267 2087 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.001s
I20260923 20:08:45.069471 2087 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790194121577027-1706-0/minicluster-data/ts-2/data,/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790194121577027-1706-0/minicluster-data/ts-2/wal
uuid: "9c6dee09bc4242438716b3e97e15c0de"
format_stamp: "Formatted at 2026-09-23 20:08:45 on dist-test-slave-th4g"
server_key: "0c8c9e6041978c72621766006e52ee26"
server_key_iv: "99325d1c945118dbd40039ae313875cf"
server_key_version: "encryptionkey@0"
I20260923 20:08:45.069700 2087 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790194121577027-1706-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790194121577027-1706-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790194121577027-1706-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:08:45.091849 1947 heartbeater.cc:503] Master 127.1.170.190:40663 was elected leader, sending a full tablet report...
I20260923 20:08:45.095343 2087 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:08:45.100348 2087 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:08:45.100597 2087 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:08:45.101547 2087 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 20:08:45.102927 2087 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 20:08:45.103005 2087 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 20:08:45.103101 2087 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 20:08:45.103145 2087 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 20:08:45.155304 2087 rpc_server.cc:307] RPC server started. Bound to: 127.1.170.131:35069
I20260923 20:08:45.155469 2214 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.170.131:35069 every 8 connection(s)
I20260923 20:08:45.156931 2087 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790194121577027-1706-0/minicluster-data/ts-2/data/info.pb
I20260923 20:08:45.165639 1706 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqUuku6/build/asan/bin/kudu as pid 2087
I20260923 20:08:45.165972 1706 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790194121577027-1706-0/minicluster-data/ts-2/wal/instance
I20260923 20:08:45.166558 1706 external_mini_cluster.cc:1428] Setting key 26a6b44a6bbda658483d4c2a4478c40c
Sep 23 20:08:45 dist-test-slave-th4g krb5kdc[1713](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194125, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.170.131@KRBTEST.COM for kudu/127.1.170.190@KRBTEST.COM
I20260923 20:08:45.185317 2217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:08:45.160445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.170.131:39305 (local address 127.1.170.190:40663)
0923 20:08:45.161285 (+ 840us) server_negotiation.cc:206] Beginning negotiation
0923 20:08:45.161290 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 20:08:45.163066 (+ 1776us) server_negotiation.cc:407] Connection header received
0923 20:08:45.165728 (+ 2662us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:08:45.165734 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:08:45.165836 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:08:45.166021 (+ 185us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:08:45.168171 (+ 2150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:08:45.169459 (+ 1288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:08:45.171727 (+ 2268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:08:45.172286 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:08:45.172432 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:08:45.177160 (+ 4728us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:08:45.177165 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:08:45.177169 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:08:45.177202 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:08:45.180844 (+ 3642us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:08:45.181988 (+ 1144us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:08:45.181998 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:08:45.182001 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:08:45.182194 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:08:45.182774 (+ 580us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:08:45.182777 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:08:45.182780 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:08:45.183094 (+ 314us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:08:45.183574 (+ 480us) server_negotiation.cc:1035] Waiting for connection context
0923 20:08:45.184419 (+ 845us) server_negotiation.cc:299] Negotiation successful
0923 20:08:45.184633 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":727,"thread_start_us":544,"threads_started":1}
I20260923 20:08:45.186647 2218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:08:45.161269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.190:40663 (local address 127.1.170.131:39305)
0923 20:08:45.162838 (+ 1569us) negotiation.cc:107] Waiting for socket to connect
0923 20:08:45.162935 (+ 97us) client_negotiation.cc:174] Beginning negotiation
0923 20:08:45.165363 (+ 2428us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:08:45.166312 (+ 949us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:08:45.166321 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:08:45.167074 (+ 753us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:08:45.167993 (+ 919us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:08:45.168014 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:08:45.169649 (+ 1635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:08:45.169653 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:08:45.171560 (+ 1907us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:08:45.171573 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:08:45.172619 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:08:45.172628 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:08:45.172934 (+ 306us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:08:45.173798 (+ 864us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:08:45.173822 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:08:45.176904 (+ 3082us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:08:45.181057 (+ 4153us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:08:45.181065 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:08:45.181081 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:08:45.181791 (+ 710us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:08:45.182312 (+ 521us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:08:45.182319 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:08:45.182323 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:08:45.182601 (+ 278us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:08:45.183547 (+ 946us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:08:45.183564 (+ 17us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:08:45.184029 (+ 465us) client_negotiation.cc:769] Sending connection context
0923 20:08:45.184330 (+ 301us) client_negotiation.cc:240] Negotiation successful
0923 20:08:45.185030 (+ 700us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1294,"thread_start_us":1000,"threads_started":1}
I20260923 20:08:45.188395 2215 heartbeater.cc:347] Connected to a master server at 127.1.170.190:40663
I20260923 20:08:45.188719 2215 heartbeater.cc:464] Registering TS with master...
I20260923 20:08:45.189661 2215 heartbeater.cc:511] Master 127.1.170.190:40663 requested a full tablet report, sending...
I20260923 20:08:45.191471 1758 ts_manager.cc:194] Registered new tserver with Master: 9c6dee09bc4242438716b3e97e15c0de (127.1.170.131:35069)
I20260923 20:08:45.192521 1758 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.170.131@KRBTEST.COM'} at 127.1.170.131:39305
I20260923 20:08:45.196880 1706 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 23 20:08:45 dist-test-slave-th4g krb5kdc[1713](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194125, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 23 20:08:45 dist-test-slave-th4g krb5kdc[1713](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194125, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.1.170.129@KRBTEST.COM
I20260923 20:08:45.259753 2223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:08:45.237457 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35080 (local address 127.1.170.129:35167)
0923 20:08:45.238487 (+ 1030us) server_negotiation.cc:206] Beginning negotiation
0923 20:08:45.238494 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:08:45.238695 (+ 201us) server_negotiation.cc:407] Connection header received
0923 20:08:45.238858 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:08:45.238865 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:08:45.239083 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:08:45.239226 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:08:45.241594 (+ 2368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:08:45.242715 (+ 1121us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:08:45.244429 (+ 1714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:08:45.245003 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:08:45.245126 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:08:45.251167 (+ 6041us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:08:45.251184 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:08:45.251200 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:08:45.251263 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:08:45.255889 (+ 4626us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:08:45.256873 (+ 984us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:08:45.256881 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:08:45.256883 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:08:45.257090 (+ 207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:08:45.257538 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:08:45.257545 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:08:45.257548 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:08:45.258077 (+ 529us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:08:45.258498 (+ 421us) server_negotiation.cc:1035] Waiting for connection context
0923 20:08:45.258861 (+ 363us) server_negotiation.cc:299] Negotiation successful
0923 20:08:45.259223 (+ 362us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":840,"thread_start_us":575,"threads_started":1}
W20260923 20:08:45.261220 1862 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35080
Sep 23 20:08:45 dist-test-slave-th4g krb5kdc[1713](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194125, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 23 20:08:45 dist-test-slave-th4g krb5kdc[1713](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194125, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.170.129@KRBTEST.COM
I20260923 20:08:45.317325 2223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:08:45.299158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35088 (local address 127.1.170.129:35167)
0923 20:08:45.299390 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0923 20:08:45.299399 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 20:08:45.299583 (+ 184us) server_negotiation.cc:407] Connection header received
0923 20:08:45.299857 (+ 274us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:08:45.299864 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:08:45.299987 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:08:45.300132 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:08:45.301609 (+ 1477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:08:45.302572 (+ 963us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:08:45.303980 (+ 1408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:08:45.304453 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:08:45.304564 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:08:45.310233 (+ 5669us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:08:45.310238 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:08:45.310242 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:08:45.310285 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:08:45.314156 (+ 3871us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:08:45.315016 (+ 860us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:08:45.315023 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:08:45.315026 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:08:45.315226 (+ 200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:08:45.315672 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:08:45.315683 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:08:45.315685 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:08:45.315988 (+ 303us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:08:45.316323 (+ 335us) server_negotiation.cc:1035] Waiting for connection context
0923 20:08:45.316739 (+ 416us) server_negotiation.cc:299] Negotiation successful
0923 20:08:45.316957 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97}
I20260923 20:08:45.319656 1706 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskqUuku6/build/asan/bin/kudu pid 1819
I20260923 20:08:45.341415 2223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:08:45.321096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35100 (local address 127.1.170.129:35167)
0923 20:08:45.321409 (+ 313us) server_negotiation.cc:206] Beginning negotiation
0923 20:08:45.321415 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:08:45.321721 (+ 306us) server_negotiation.cc:407] Connection header received
0923 20:08:45.321820 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:08:45.321823 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:08:45.321960 (+ 137us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:08:45.322052 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:08:45.323892 (+ 1840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:08:45.324879 (+ 987us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:08:45.326848 (+ 1969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:08:45.327469 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:08:45.327576 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:08:45.331865 (+ 4289us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:08:45.331870 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:08:45.331873 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:08:45.331913 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:08:45.337153 (+ 5240us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:08:45.338517 (+ 1364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:08:45.338531 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:08:45.338535 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:08:45.338933 (+ 398us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:08:45.339509 (+ 576us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:08:45.339515 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:08:45.339526 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:08:45.340091 (+ 565us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:08:45.340498 (+ 407us) server_negotiation.cc:1035] Waiting for connection context
0923 20:08:45.340740 (+ 242us) server_negotiation.cc:299] Negotiation successful
0923 20:08:45.340934 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124}
I20260923 20:08:45.342552 1942 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:08:45.432416 1706 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqUuku6/build/asan/bin/kudu with pid 1819
I20260923 20:08:45.448282 1706 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskqUuku6/build/asan/bin/kudu pid 1953
Sep 23 20:08:45 dist-test-slave-th4g krb5kdc[1713](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194125, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.170.130@KRBTEST.COM
I20260923 20:08:45.467244 2230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:08:45.449500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60208 (local address 127.1.170.130:44139)
0923 20:08:45.450103 (+ 603us) server_negotiation.cc:206] Beginning negotiation
0923 20:08:45.450111 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 20:08:45.450145 (+ 34us) server_negotiation.cc:407] Connection header received
0923 20:08:45.450241 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:08:45.450248 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:08:45.450640 (+ 392us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:08:45.450923 (+ 283us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:08:45.452310 (+ 1387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:08:45.453834 (+ 1524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:08:45.455606 (+ 1772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:08:45.456178 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:08:45.456318 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:08:45.460762 (+ 4444us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:08:45.460775 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:08:45.460791 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:08:45.460830 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:08:45.464210 (+ 3380us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:08:45.465050 (+ 840us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:08:45.465058 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:08:45.465060 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:08:45.465205 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:08:45.465554 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:08:45.465557 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:08:45.465559 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:08:45.465990 (+ 431us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:08:45.466223 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0923 20:08:45.466697 (+ 474us) server_negotiation.cc:299] Negotiation successful
0923 20:08:45.466901 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":422,"thread_start_us":255,"threads_started":1}
I20260923 20:08:45.468847 2076 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:08:45.554875 1706 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqUuku6/build/asan/bin/kudu with pid 1953
I20260923 20:08:45.571811 1706 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskqUuku6/build/asan/bin/kudu pid 2087
Sep 23 20:08:45 dist-test-slave-th4g krb5kdc[1713](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194125, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.170.131@KRBTEST.COM
I20260923 20:08:45.591738 2232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:08:45.572909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60104 (local address 127.1.170.131:35069)
0923 20:08:45.573444 (+ 535us) server_negotiation.cc:206] Beginning negotiation
0923 20:08:45.573451 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:08:45.573481 (+ 30us) server_negotiation.cc:407] Connection header received
0923 20:08:45.573540 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:08:45.573546 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:08:45.573757 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:08:45.574012 (+ 255us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:08:45.575350 (+ 1338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:08:45.576400 (+ 1050us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:08:45.577838 (+ 1438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:08:45.578727 (+ 889us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:08:45.578865 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:08:45.583950 (+ 5085us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:08:45.583971 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:08:45.584001 (+ 30us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:08:45.584064 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:08:45.587747 (+ 3683us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:08:45.588797 (+ 1050us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:08:45.588805 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:08:45.588808 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:08:45.588975 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:08:45.589664 (+ 689us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:08:45.589670 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:08:45.589673 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:08:45.590338 (+ 665us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:08:45.590584 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0923 20:08:45.591065 (+ 481us) server_negotiation.cc:299] Negotiation successful
0923 20:08:45.591301 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":343,"thread_start_us":220,"threads_started":1}
I20260923 20:08:45.593400 2210 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:08:45.676959 1706 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqUuku6/build/asan/bin/kudu with pid 2087
I20260923 20:08:45.693545 1706 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskqUuku6/build/asan/bin/kudu pid 1726
I20260923 20:08:45.694753 1788 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:08:45.800093 1706 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqUuku6/build/asan/bin/kudu with pid 1726
2026-09-23T20:08:45Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (4160 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790194121577027-1706-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 20:08:45 dist-test-slave-th4g krb5kdc[2237](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 20:08:45 dist-test-slave-th4g krb5kdc[2237](info): set up 2 sockets
Sep 23 20:08:45 dist-test-slave-th4g krb5kdc[2237](info): commencing operation
krb5kdc: starting...
W20260923 20:08:46.906128 1706 mini_kdc.cc:121] Time spent starting KDC: real 1.062s user 0.000s sys 0.015s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790194121577027-1706-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790194121577027-1706-0/krb5kdc/test-user.keytab.
Sep 23 20:08:47 dist-test-slave-th4g krb5kdc[2237](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194127, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T20:08:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T20:08:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.170.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.170.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.170.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790194121577027-1706-0/krb5kdc/kudu_127.1.170.190.keytab.
Entry for principal kudu/127.1.170.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790194121577027-1706-0/krb5kdc/kudu_127.1.170.190.keytab.
I20260923 20:08:47.095052 1706 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqUuku6/build/asan/bin/kudu
/tmp/dist-test-taskqUuku6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790194121577027-1706-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790194121577027-1706-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790194121577027-1706-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790194121577027-1706-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.170.190:40377
--webserver_interface=127.1.170.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790194121577027-1706-0/krb5kdc/kudu_127.1.170.190.keytab
--principal=kudu/127.1.170.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.170.148:46869
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.170.190:40377
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790194121577027-1706-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790194121577027-1706-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790194121577027-1706-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:08:47.418615 2250 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:08:47.418965 2250 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:08:47.419130 2250 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:08:47.426442 2250 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 20:08:47.426606 2250 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:08:47.426715 2250 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:08:47.426822 2250 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 20:08:47.426903 2250 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 20:08:47.436996 2250 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.170.148:46869
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790194121577027-1706-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790194121577027-1706-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.170.190:40377
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790194121577027-1706-0/krb5kdc/kudu_127.1.170.190.keytab
--principal=kudu/127.1.170.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.170.190:40377
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790194121577027-1706-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.170.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790194121577027-1706-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:08:47.438441 2250 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:08:47.439709 2250 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:08:47.451321 2256 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:08:47.451260 2255 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:08:47.451334 2258 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:08:47.452729 2250 server_base.cc:1034] running on GCE node
I20260923 20:08:47.453557 2250 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:08:47.455144 2250 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:08:47.456462 2250 hybrid_clock.cc:648] HybridClock initialized: now 1790194127456337 us; error 52 us; skew 500 ppm
I20260923 20:08:47.456921 2250 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:08:47 dist-test-slave-th4g krb5kdc[2237](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194127, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.170.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:08:47.464970 2250 init.cc:377] Logged in from keytab as kudu/127.1.170.190@KRBTEST.COM (short username kudu)
I20260923 20:08:47.466814 2250 webserver.cc:460] Webserver started at http://127.1.170.190:32909/ using document root <none> and password file <none>
I20260923 20:08:47.467525 2250 fs_manager.cc:359] Metadata directory not provided
I20260923 20:08:47.467628 2250 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:08:47.467887 2250 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:08:47.470942 2250 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790194121577027-1706-0/minicluster-data/master-0/data/instance:
uuid: "7bae267bb520442593016cf0b83359cb"
format_stamp: "Formatted at 2026-09-23 20:08:47 on dist-test-slave-th4g"
server_key: "0f1ed12bdddfe0b7cd61bfcc46f83a12"
server_key_iv: "c164a28528933500d31845c469c746fa"
server_key_version: "encryptionkey@0"
I20260923 20:08:47.471738 2250 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790194121577027-1706-0/minicluster-data/master-0/wal/instance:
uuid: "7bae267bb520442593016cf0b83359cb"
format_stamp: "Formatted at 2026-09-23 20:08:47 on dist-test-slave-th4g"
server_key: "0f1ed12bdddfe0b7cd61bfcc46f83a12"
server_key_iv: "c164a28528933500d31845c469c746fa"
server_key_version: "encryptionkey@0"
I20260923 20:08:47.477415 2250 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260923 20:08:47.481093 2265 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:08:47.482705 2250 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 20:08:47.482894 2250 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790194121577027-1706-0/minicluster-data/master-0/data,/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790194121577027-1706-0/minicluster-data/master-0/wal
uuid: "7bae267bb520442593016cf0b83359cb"
format_stamp: "Formatted at 2026-09-23 20:08:47 on dist-test-slave-th4g"
server_key: "0f1ed12bdddfe0b7cd61bfcc46f83a12"
server_key_iv: "c164a28528933500d31845c469c746fa"
server_key_version: "encryptionkey@0"
I20260923 20:08:47.483089 2250 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790194121577027-1706-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790194121577027-1706-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790194121577027-1706-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:08:47.497099 2250 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:08:47.501173 2250 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:08:47.501473 2250 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:08:47.519797 2250 rpc_server.cc:307] RPC server started. Bound to: 127.1.170.190:40377
I20260923 20:08:47.519826 2316 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.170.190:40377 every 8 connection(s)
I20260923 20:08:47.521162 2250 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790194121577027-1706-0/minicluster-data/master-0/data/info.pb
I20260923 20:08:47.522653 1706 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqUuku6/build/asan/bin/kudu as pid 2250
I20260923 20:08:47.522900 1706 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790194121577027-1706-0/minicluster-data/master-0/wal/instance
I20260923 20:08:47.523729 1706 external_mini_cluster.cc:1428] Setting key 2534fb01f7f5ca9de74b95e66cd21038
I20260923 20:08:47.526320 2317 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 23 20:08:47 dist-test-slave-th4g krb5kdc[2237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194127, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.170.190@KRBTEST.COM
I20260923 20:08:47.539971 2317 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7bae267bb520442593016cf0b83359cb: Bootstrap starting.
I20260923 20:08:47.543319 2317 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7bae267bb520442593016cf0b83359cb: Neither blocks nor log segments found. Creating new log.
I20260923 20:08:47.544430 2317 log.cc:824] T 00000000000000000000000000000000 P 7bae267bb520442593016cf0b83359cb: Log is configured to *not* fsync() on all Append() calls
I20260923 20:08:47.547643 2317 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7bae267bb520442593016cf0b83359cb: No bootstrap required, opened a new log
I20260923 20:08:47.547618 2320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:08:47.525776 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48366 (local address 127.1.170.190:40377)
0923 20:08:47.526775 (+ 999us) server_negotiation.cc:206] Beginning negotiation
0923 20:08:47.526786 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0923 20:08:47.526826 (+ 40us) server_negotiation.cc:407] Connection header received
0923 20:08:47.528453 (+ 1627us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:08:47.528473 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:08:47.528865 (+ 392us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:08:47.529204 (+ 339us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:08:47.531030 (+ 1826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:08:47.532413 (+ 1383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:08:47.534277 (+ 1864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:08:47.535015 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:08:47.535167 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:08:47.539895 (+ 4728us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:08:47.539905 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:08:47.539923 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:08:47.539966 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:08:47.543214 (+ 3248us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:08:47.544017 (+ 803us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:08:47.544024 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:08:47.544030 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:08:47.544198 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:08:47.544588 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:08:47.544593 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:08:47.544598 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:08:47.545155 (+ 557us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:08:47.545513 (+ 358us) server_negotiation.cc:1035] Waiting for connection context
0923 20:08:47.546169 (+ 656us) server_negotiation.cc:299] Negotiation successful
0923 20:08:47.546603 (+ 434us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":796,"thread_start_us":525,"threads_started":1}
I20260923 20:08:47.555783 2317 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7bae267bb520442593016cf0b83359cb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7bae267bb520442593016cf0b83359cb" member_type: VOTER last_known_addr { host: "127.1.170.190" port: 40377 } }
I20260923 20:08:47.556277 2317 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7bae267bb520442593016cf0b83359cb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:08:47.556408 2317 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7bae267bb520442593016cf0b83359cb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7bae267bb520442593016cf0b83359cb, State: Initialized, Role: FOLLOWER
I20260923 20:08:47.557039 2317 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7bae267bb520442593016cf0b83359cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7bae267bb520442593016cf0b83359cb" member_type: VOTER last_known_addr { host: "127.1.170.190" port: 40377 } }
I20260923 20:08:47.557301 2317 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7bae267bb520442593016cf0b83359cb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 20:08:47.557459 2317 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7bae267bb520442593016cf0b83359cb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 20:08:47.557626 2317 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7bae267bb520442593016cf0b83359cb [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:08:47.559844 2317 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7bae267bb520442593016cf0b83359cb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7bae267bb520442593016cf0b83359cb" member_type: VOTER last_known_addr { host: "127.1.170.190" port: 40377 } }
I20260923 20:08:47.560360 2317 leader_election.cc:302] T 00000000000000000000000000000000 P 7bae267bb520442593016cf0b83359cb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7bae267bb520442593016cf0b83359cb; no voters:
I20260923 20:08:47.560971 2317 leader_election.cc:288] T 00000000000000000000000000000000 P 7bae267bb520442593016cf0b83359cb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 20:08:47.561074 2322 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7bae267bb520442593016cf0b83359cb [term 1 FOLLOWER]: Leader election won for term 1
I20260923 20:08:47.562242 2322 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7bae267bb520442593016cf0b83359cb [term 1 LEADER]: Becoming Leader. State: Replica: 7bae267bb520442593016cf0b83359cb, State: Running, Role: LEADER
I20260923 20:08:47.562788 2322 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7bae267bb520442593016cf0b83359cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7bae267bb520442593016cf0b83359cb" member_type: VOTER last_known_addr { host: "127.1.170.190" port: 40377 } }
I20260923 20:08:47.563244 2317 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7bae267bb520442593016cf0b83359cb [sys.catalog]: configured and running, proceeding with master startup.
I20260923 20:08:47.565757 2323 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7bae267bb520442593016cf0b83359cb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7bae267bb520442593016cf0b83359cb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7bae267bb520442593016cf0b83359cb" member_type: VOTER last_known_addr { host: "127.1.170.190" port: 40377 } } }
I20260923 20:08:47.565737 2324 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7bae267bb520442593016cf0b83359cb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7bae267bb520442593016cf0b83359cb. Latest consensus state: current_term: 1 leader_uuid: "7bae267bb520442593016cf0b83359cb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7bae267bb520442593016cf0b83359cb" member_type: VOTER last_known_addr { host: "127.1.170.190" port: 40377 } } }
I20260923 20:08:47.566205 2323 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7bae267bb520442593016cf0b83359cb [sys.catalog]: This master's current role is: LEADER
I20260923 20:08:47.566205 2324 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7bae267bb520442593016cf0b83359cb [sys.catalog]: This master's current role is: LEADER
I20260923 20:08:47.569100 2329 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 20:08:47.574702 2329 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 20:08:47.584299 2329 catalog_manager.cc:1347] Generated new cluster ID: 0ccee7c8c2fb4b83af8e4d8ec7fa4f0c
I20260923 20:08:47.584419 2329 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 20:08:47.601567 2329 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 20:08:47.603024 2329 catalog_manager.cc:1504] Loading token signing keys...
I20260923 20:08:47.615180 2329 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7bae267bb520442593016cf0b83359cb: Generated new TSK 0
I20260923 20:08:47.616317 2329 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.170.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.170.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.170.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790194121577027-1706-0/krb5kdc/kudu_127.1.170.129.keytab.
Entry for principal kudu/127.1.170.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790194121577027-1706-0/krb5kdc/kudu_127.1.170.129.keytab.
I20260923 20:08:47.670416 1706 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqUuku6/build/asan/bin/kudu
/tmp/dist-test-taskqUuku6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790194121577027-1706-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790194121577027-1706-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790194121577027-1706-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790194121577027-1706-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.170.129:0
--local_ip_for_outbound_sockets=127.1.170.129
--webserver_interface=127.1.170.129
--webserver_port=0
--tserver_master_addrs=127.1.170.190:40377
--keytab_file=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790194121577027-1706-0/krb5kdc/kudu_127.1.170.129.keytab
--principal=kudu/127.1.170.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.170.148:46869
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790194121577027-1706-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790194121577027-1706-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790194121577027-1706-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:08:48.018738 2343 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:08:48.019076 2343 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:08:48.019237 2343 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:08:48.026702 2343 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:08:48.026832 2343 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:08:48.026953 2343 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.170.129
I20260923 20:08:48.037173 2343 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.170.148:46869
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790194121577027-1706-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790194121577027-1706-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790194121577027-1706-0/krb5kdc/kudu_127.1.170.129.keytab
--principal=kudu/127.1.170.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.170.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790194121577027-1706-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.170.129
--webserver_port=0
--tserver_master_addrs=127.1.170.190:40377
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.170.129
--log_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790194121577027-1706-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:08:48.038580 2343 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:08:48.039945 2343 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:08:48.053828 2351 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:08:48.053898 2348 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:08:48.053846 2349 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:08:48.055217 2343 server_base.cc:1034] running on GCE node
I20260923 20:08:48.055850 2343 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:08:48.056847 2343 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:08:48.058200 2343 hybrid_clock.cc:648] HybridClock initialized: now 1790194128058133 us; error 47 us; skew 500 ppm
I20260923 20:08:48.058555 2343 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:08:48 dist-test-slave-th4g krb5kdc[2237](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194128, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.170.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:08:48.061789 2343 init.cc:377] Logged in from keytab as kudu/127.1.170.129@KRBTEST.COM (short username kudu)
I20260923 20:08:48.063375 2343 webserver.cc:460] Webserver started at http://127.1.170.129:36055/ using document root <none> and password file <none>
I20260923 20:08:48.064054 2343 fs_manager.cc:359] Metadata directory not provided
I20260923 20:08:48.064201 2343 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:08:48.064538 2343 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:08:48.067520 2343 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790194121577027-1706-0/minicluster-data/ts-0/data/instance:
uuid: "4de9ea06b2704f4ca3fc1cd50ddd6e74"
format_stamp: "Formatted at 2026-09-23 20:08:48 on dist-test-slave-th4g"
server_key: "d1e05810d3ca249dbe0c68e404b02d2b"
server_key_iv: "9023b3744036d2a3341bf05f367bfd11"
server_key_version: "encryptionkey@0"
I20260923 20:08:48.068295 2343 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790194121577027-1706-0/minicluster-data/ts-0/wal/instance:
uuid: "4de9ea06b2704f4ca3fc1cd50ddd6e74"
format_stamp: "Formatted at 2026-09-23 20:08:48 on dist-test-slave-th4g"
server_key: "d1e05810d3ca249dbe0c68e404b02d2b"
server_key_iv: "9023b3744036d2a3341bf05f367bfd11"
server_key_version: "encryptionkey@0"
I20260923 20:08:48.073516 2343 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260923 20:08:48.077313 2358 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:08:48.079057 2343 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 20:08:48.079306 2343 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790194121577027-1706-0/minicluster-data/ts-0/data,/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790194121577027-1706-0/minicluster-data/ts-0/wal
uuid: "4de9ea06b2704f4ca3fc1cd50ddd6e74"
format_stamp: "Formatted at 2026-09-23 20:08:48 on dist-test-slave-th4g"
server_key: "d1e05810d3ca249dbe0c68e404b02d2b"
server_key_iv: "9023b3744036d2a3341bf05f367bfd11"
server_key_version: "encryptionkey@0"
I20260923 20:08:48.079558 2343 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790194121577027-1706-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790194121577027-1706-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790194121577027-1706-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:08:48.092322 2343 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:08:48.096752 2343 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:08:48.097084 2343 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:08:48.098109 2343 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 20:08:48.099714 2343 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 20:08:48.099845 2343 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 20:08:48.100006 2343 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 20:08:48.100083 2343 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 20:08:48.136284 2343 rpc_server.cc:307] RPC server started. Bound to: 127.1.170.129:44597
I20260923 20:08:48.136374 2470 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.170.129:44597 every 8 connection(s)
I20260923 20:08:48.137615 2343 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790194121577027-1706-0/minicluster-data/ts-0/data/info.pb
I20260923 20:08:48.141714 1706 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqUuku6/build/asan/bin/kudu as pid 2343
I20260923 20:08:48.142010 1706 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790194121577027-1706-0/minicluster-data/ts-0/wal/instance
I20260923 20:08:48.142792 1706 external_mini_cluster.cc:1428] Setting key fbca723af9e00eb7942642ce2e9a0701
Sep 23 20:08:48 dist-test-slave-th4g krb5kdc[2237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194128, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.170.129@KRBTEST.COM for kudu/127.1.170.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.1.170.130@KRBTEST.COM; defaulting to no policy
I20260923 20:08:48.160075 2473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:08:48.140311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.170.129:57039 (local address 127.1.170.190:40377)
0923 20:08:48.140731 (+ 420us) server_negotiation.cc:206] Beginning negotiation
0923 20:08:48.140736 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 20:08:48.141821 (+ 1085us) server_negotiation.cc:407] Connection header received
0923 20:08:48.144127 (+ 2306us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:08:48.144133 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:08:48.144201 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:08:48.144311 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:08:48.146489 (+ 2178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:08:48.147172 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:08:48.149700 (+ 2528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:08:48.150182 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:08:48.150321 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:08:48.153715 (+ 3394us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:08:48.153720 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:08:48.153724 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:08:48.153756 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:08:48.156721 (+ 2965us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:08:48.157719 (+ 998us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:08:48.157723 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:08:48.157726 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:08:48.157845 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:08:48.158334 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:08:48.158342 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:08:48.158346 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:08:48.158611 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:08:48.158782 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0923 20:08:48.159497 (+ 715us) server_negotiation.cc:299] Negotiation successful
0923 20:08:48.159722 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":309,"thread_start_us":197,"threads_started":1}
I20260923 20:08:48.161327 2474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:08:48.140849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.190:40377 (local address 127.1.170.129:57039)
0923 20:08:48.141631 (+ 782us) negotiation.cc:107] Waiting for socket to connect
0923 20:08:48.141682 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0923 20:08:48.143880 (+ 2198us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:08:48.144619 (+ 739us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:08:48.144628 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:08:48.145439 (+ 811us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:08:48.146296 (+ 857us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:08:48.146327 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:08:48.147344 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:08:48.147353 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:08:48.149533 (+ 2180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:08:48.149551 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:08:48.150410 (+ 859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:08:48.150419 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:08:48.150654 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:08:48.151343 (+ 689us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:08:48.151365 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:08:48.153468 (+ 2103us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:08:48.156921 (+ 3453us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:08:48.156937 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:08:48.156953 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:08:48.157537 (+ 584us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:08:48.157989 (+ 452us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:08:48.158004 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:08:48.158008 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:08:48.158212 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:08:48.158755 (+ 543us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:08:48.158764 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:08:48.159129 (+ 365us) client_negotiation.cc:769] Sending connection context
0923 20:08:48.159409 (+ 280us) client_negotiation.cc:240] Negotiation successful
0923 20:08:48.159978 (+ 569us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":494,"thread_start_us":253,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.170.130@KRBTEST.COM" created.
I20260923 20:08:48.163468 2471 heartbeater.cc:347] Connected to a master server at 127.1.170.190:40377
I20260923 20:08:48.163887 2471 heartbeater.cc:464] Registering TS with master...
I20260923 20:08:48.164685 2471 heartbeater.cc:511] Master 127.1.170.190:40377 requested a full tablet report, sending...
I20260923 20:08:48.166970 2282 ts_manager.cc:194] Registered new tserver with Master: 4de9ea06b2704f4ca3fc1cd50ddd6e74 (127.1.170.129:44597)
I20260923 20:08:48.168437 2282 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.170.129@KRBTEST.COM'} at 127.1.170.129:57039
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.170.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790194121577027-1706-0/krb5kdc/kudu_127.1.170.130.keytab.
Entry for principal kudu/127.1.170.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790194121577027-1706-0/krb5kdc/kudu_127.1.170.130.keytab.
I20260923 20:08:48.189600 1706 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqUuku6/build/asan/bin/kudu
/tmp/dist-test-taskqUuku6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790194121577027-1706-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790194121577027-1706-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790194121577027-1706-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790194121577027-1706-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.170.130:0
--local_ip_for_outbound_sockets=127.1.170.130
--webserver_interface=127.1.170.130
--webserver_port=0
--tserver_master_addrs=127.1.170.190:40377
--keytab_file=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790194121577027-1706-0/krb5kdc/kudu_127.1.170.130.keytab
--principal=kudu/127.1.170.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.170.148:46869
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790194121577027-1706-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790194121577027-1706-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790194121577027-1706-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:08:48.512895 2477 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:08:48.513288 2477 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:08:48.513471 2477 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:08:48.521176 2477 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:08:48.521328 2477 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:08:48.521512 2477 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.170.130
I20260923 20:08:48.532048 2477 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.170.148:46869
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790194121577027-1706-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790194121577027-1706-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790194121577027-1706-0/krb5kdc/kudu_127.1.170.130.keytab
--principal=kudu/127.1.170.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.170.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790194121577027-1706-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.170.130
--webserver_port=0
--tserver_master_addrs=127.1.170.190:40377
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.170.130
--log_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790194121577027-1706-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:08:48.533352 2477 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:08:48.534772 2477 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:08:48.548682 2482 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:08:48.548709 2483 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:08:48.548733 2485 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:08:48.549562 2477 server_base.cc:1034] running on GCE node
I20260923 20:08:48.550257 2477 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:08:48.551275 2477 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:08:48.552496 2477 hybrid_clock.cc:648] HybridClock initialized: now 1790194128552427 us; error 56 us; skew 500 ppm
I20260923 20:08:48.552868 2477 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:08:48 dist-test-slave-th4g krb5kdc[2237](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194128, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.170.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:08:48.556418 2477 init.cc:377] Logged in from keytab as kudu/127.1.170.130@KRBTEST.COM (short username kudu)
I20260923 20:08:48.558167 2477 webserver.cc:460] Webserver started at http://127.1.170.130:39445/ using document root <none> and password file <none>
I20260923 20:08:48.558883 2477 fs_manager.cc:359] Metadata directory not provided
I20260923 20:08:48.558986 2477 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:08:48.559345 2477 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:08:48.562317 2477 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790194121577027-1706-0/minicluster-data/ts-1/data/instance:
uuid: "0791199741904b30aeb9576806fcc663"
format_stamp: "Formatted at 2026-09-23 20:08:48 on dist-test-slave-th4g"
server_key: "8f3046b815c97b0cfea8d173a38d932c"
server_key_iv: "8d54fc9f5746327091e7e5be712735cf"
server_key_version: "encryptionkey@0"
I20260923 20:08:48.563133 2477 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790194121577027-1706-0/minicluster-data/ts-1/wal/instance:
uuid: "0791199741904b30aeb9576806fcc663"
format_stamp: "Formatted at 2026-09-23 20:08:48 on dist-test-slave-th4g"
server_key: "8f3046b815c97b0cfea8d173a38d932c"
server_key_iv: "8d54fc9f5746327091e7e5be712735cf"
server_key_version: "encryptionkey@0"
I20260923 20:08:48.568549 2477 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260923 20:08:48.572259 2492 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:08:48.574014 2477 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260923 20:08:48.574188 2477 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790194121577027-1706-0/minicluster-data/ts-1/data,/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790194121577027-1706-0/minicluster-data/ts-1/wal
uuid: "0791199741904b30aeb9576806fcc663"
format_stamp: "Formatted at 2026-09-23 20:08:48 on dist-test-slave-th4g"
server_key: "8f3046b815c97b0cfea8d173a38d932c"
server_key_iv: "8d54fc9f5746327091e7e5be712735cf"
server_key_version: "encryptionkey@0"
I20260923 20:08:48.574404 2477 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790194121577027-1706-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790194121577027-1706-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790194121577027-1706-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:08:48.585778 2477 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:08:48.590430 2477 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:08:48.590749 2477 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:08:48.591755 2477 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 20:08:48.593133 2477 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 20:08:48.593214 2477 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 20:08:48.593386 2477 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 20:08:48.593436 2477 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 20:08:48.627998 2477 rpc_server.cc:307] RPC server started. Bound to: 127.1.170.130:45725
I20260923 20:08:48.628046 2604 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.170.130:45725 every 8 connection(s)
I20260923 20:08:48.629343 2477 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790194121577027-1706-0/minicluster-data/ts-1/data/info.pb
I20260923 20:08:48.637378 1706 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqUuku6/build/asan/bin/kudu as pid 2477
I20260923 20:08:48.637583 1706 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790194121577027-1706-0/minicluster-data/ts-1/wal/instance
I20260923 20:08:48.638078 1706 external_mini_cluster.cc:1428] Setting key a51a6c923fe35126d482fb5989a7b906
Sep 23 20:08:48 dist-test-slave-th4g krb5kdc[2237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194128, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.170.130@KRBTEST.COM for kudu/127.1.170.190@KRBTEST.COM
I20260923 20:08:48.652725 2473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:08:48.632141 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.170.130:43621 (local address 127.1.170.190:40377)
0923 20:08:48.632344 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0923 20:08:48.632349 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 20:08:48.633542 (+ 1193us) server_negotiation.cc:407] Connection header received
0923 20:08:48.635377 (+ 1835us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:08:48.635382 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:08:48.635476 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:08:48.635592 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:08:48.637228 (+ 1636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:08:48.638113 (+ 885us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:08:48.640085 (+ 1972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:08:48.640668 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:08:48.640802 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:08:48.645277 (+ 4475us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:08:48.645287 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:08:48.645296 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:08:48.645331 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:08:48.649063 (+ 3732us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:08:48.650070 (+ 1007us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:08:48.650078 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:08:48.650085 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:08:48.650241 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:08:48.650707 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:08:48.650715 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:08:48.650721 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:08:48.651123 (+ 402us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:08:48.651278 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0923 20:08:48.652027 (+ 749us) server_negotiation.cc:299] Negotiation successful
0923 20:08:48.652283 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260923 20:08:48.653573 2607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:08:48.632641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.190:40377 (local address 127.1.170.130:43621)
0923 20:08:48.633376 (+ 735us) negotiation.cc:107] Waiting for socket to connect
0923 20:08:48.633422 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0923 20:08:48.635114 (+ 1692us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:08:48.635817 (+ 703us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:08:48.635825 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:08:48.636327 (+ 502us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:08:48.637022 (+ 695us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:08:48.637048 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:08:48.638289 (+ 1241us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:08:48.638294 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:08:48.639921 (+ 1627us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:08:48.639944 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:08:48.640893 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:08:48.640898 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:08:48.641098 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:08:48.641869 (+ 771us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:08:48.641947 (+ 78us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:08:48.644985 (+ 3038us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:08:48.649336 (+ 4351us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:08:48.649345 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:08:48.649379 (+ 34us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:08:48.649867 (+ 488us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:08:48.650384 (+ 517us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:08:48.650388 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:08:48.650390 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:08:48.650574 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:08:48.651249 (+ 675us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:08:48.651255 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:08:48.651630 (+ 375us) client_negotiation.cc:769] Sending connection context
0923 20:08:48.651868 (+ 238us) client_negotiation.cc:240] Negotiation successful
0923 20:08:48.652372 (+ 504us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":518,"thread_start_us":272,"threads_started":1}
I20260923 20:08:48.655124 2605 heartbeater.cc:347] Connected to a master server at 127.1.170.190:40377
I20260923 20:08:48.655479 2605 heartbeater.cc:464] Registering TS with master...
I20260923 20:08:48.656181 2605 heartbeater.cc:511] Master 127.1.170.190:40377 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.1.170.131@KRBTEST.COM; defaulting to no policy
I20260923 20:08:48.657752 2282 ts_manager.cc:194] Registered new tserver with Master: 0791199741904b30aeb9576806fcc663 (127.1.170.130:45725)
I20260923 20:08:48.658718 2282 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.170.130@KRBTEST.COM'} at 127.1.170.130:43621
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.170.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.170.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790194121577027-1706-0/krb5kdc/kudu_127.1.170.131.keytab.
Entry for principal kudu/127.1.170.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790194121577027-1706-0/krb5kdc/kudu_127.1.170.131.keytab.
I20260923 20:08:48.681845 1706 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqUuku6/build/asan/bin/kudu
/tmp/dist-test-taskqUuku6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790194121577027-1706-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790194121577027-1706-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790194121577027-1706-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790194121577027-1706-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.170.131:0
--local_ip_for_outbound_sockets=127.1.170.131
--webserver_interface=127.1.170.131
--webserver_port=0
--tserver_master_addrs=127.1.170.190:40377
--keytab_file=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790194121577027-1706-0/krb5kdc/kudu_127.1.170.131.keytab
--principal=kudu/127.1.170.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.170.148:46869
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790194121577027-1706-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790194121577027-1706-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790194121577027-1706-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:08:49.009387 2610 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:08:49.009682 2610 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:08:49.009779 2610 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:08:49.017241 2610 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:08:49.017381 2610 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:08:49.017555 2610 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.170.131
I20260923 20:08:49.027355 2610 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.170.148:46869
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790194121577027-1706-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790194121577027-1706-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790194121577027-1706-0/krb5kdc/kudu_127.1.170.131.keytab
--principal=kudu/127.1.170.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.170.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790194121577027-1706-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.170.131
--webserver_port=0
--tserver_master_addrs=127.1.170.190:40377
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.170.131
--log_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790194121577027-1706-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:08:49.028710 2610 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:08:49.029999 2610 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:08:49.043808 2615 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:08:49.043823 2618 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:08:49.043853 2616 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:08:49.044894 2610 server_base.cc:1034] running on GCE node
I20260923 20:08:49.045607 2610 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:08:49.046751 2610 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:08:49.048144 2610 hybrid_clock.cc:648] HybridClock initialized: now 1790194129048054 us; error 37 us; skew 500 ppm
I20260923 20:08:49.049038 2610 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:08:49 dist-test-slave-th4g krb5kdc[2237](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194129, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.170.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:08:49.052523 2610 init.cc:377] Logged in from keytab as kudu/127.1.170.131@KRBTEST.COM (short username kudu)
I20260923 20:08:49.054157 2610 webserver.cc:460] Webserver started at http://127.1.170.131:36903/ using document root <none> and password file <none>
I20260923 20:08:49.054880 2610 fs_manager.cc:359] Metadata directory not provided
I20260923 20:08:49.055032 2610 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:08:49.055353 2610 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:08:49.058349 2610 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790194121577027-1706-0/minicluster-data/ts-2/data/instance:
uuid: "d714201f8d2d4b989623b62e603c6d1e"
format_stamp: "Formatted at 2026-09-23 20:08:49 on dist-test-slave-th4g"
server_key: "1cea5fbbcdb13517413aefb6ca13248d"
server_key_iv: "f645675677131df257debc0e1afd3bf8"
server_key_version: "encryptionkey@0"
I20260923 20:08:49.059157 2610 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790194121577027-1706-0/minicluster-data/ts-2/wal/instance:
uuid: "d714201f8d2d4b989623b62e603c6d1e"
format_stamp: "Formatted at 2026-09-23 20:08:49 on dist-test-slave-th4g"
server_key: "1cea5fbbcdb13517413aefb6ca13248d"
server_key_iv: "f645675677131df257debc0e1afd3bf8"
server_key_version: "encryptionkey@0"
I20260923 20:08:49.064661 2610 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260923 20:08:49.068320 2625 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:08:49.069831 2610 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260923 20:08:49.070034 2610 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790194121577027-1706-0/minicluster-data/ts-2/data,/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790194121577027-1706-0/minicluster-data/ts-2/wal
uuid: "d714201f8d2d4b989623b62e603c6d1e"
format_stamp: "Formatted at 2026-09-23 20:08:49 on dist-test-slave-th4g"
server_key: "1cea5fbbcdb13517413aefb6ca13248d"
server_key_iv: "f645675677131df257debc0e1afd3bf8"
server_key_version: "encryptionkey@0"
I20260923 20:08:49.070221 2610 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790194121577027-1706-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790194121577027-1706-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790194121577027-1706-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:08:49.088131 2610 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:08:49.093339 2610 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:08:49.093598 2610 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:08:49.094620 2610 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 20:08:49.096015 2610 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 20:08:49.096095 2610 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 20:08:49.096194 2610 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 20:08:49.096228 2610 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 20:08:49.136116 2610 rpc_server.cc:307] RPC server started. Bound to: 127.1.170.131:39233
I20260923 20:08:49.136109 2737 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.170.131:39233 every 8 connection(s)
I20260923 20:08:49.137959 2610 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790194121577027-1706-0/minicluster-data/ts-2/data/info.pb
I20260923 20:08:49.140276 1706 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqUuku6/build/asan/bin/kudu as pid 2610
I20260923 20:08:49.140466 1706 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790194121577027-1706-0/minicluster-data/ts-2/wal/instance
I20260923 20:08:49.141016 1706 external_mini_cluster.cc:1428] Setting key 36c07591e79b1f3d6b10c59ce0390ea7
Sep 23 20:08:49 dist-test-slave-th4g krb5kdc[2237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194129, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.170.131@KRBTEST.COM for kudu/127.1.170.190@KRBTEST.COM
I20260923 20:08:49.161471 2473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:08:49.140727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.170.131:46833 (local address 127.1.170.190:40377)
0923 20:08:49.140924 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0923 20:08:49.140930 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:08:49.142903 (+ 1973us) server_negotiation.cc:407] Connection header received
0923 20:08:49.145047 (+ 2144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:08:49.145052 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:08:49.145139 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:08:49.145262 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:08:49.146811 (+ 1549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:08:49.147551 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:08:49.149103 (+ 1552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:08:49.149521 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:08:49.149680 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:08:49.154521 (+ 4841us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:08:49.154527 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:08:49.154531 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:08:49.154701 (+ 170us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:08:49.157968 (+ 3267us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:08:49.158915 (+ 947us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:08:49.158921 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:08:49.158925 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:08:49.159063 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:08:49.159583 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:08:49.159591 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:08:49.159599 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:08:49.159884 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:08:49.160052 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0923 20:08:49.160838 (+ 786us) server_negotiation.cc:299] Negotiation successful
0923 20:08:49.161081 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260923 20:08:49.162575 2740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:08:49.141277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.190:40377 (local address 127.1.170.131:46833)
0923 20:08:49.142584 (+ 1307us) negotiation.cc:107] Waiting for socket to connect
0923 20:08:49.142633 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0923 20:08:49.144784 (+ 2151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:08:49.145488 (+ 704us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:08:49.145496 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:08:49.146086 (+ 590us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:08:49.146642 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:08:49.146660 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:08:49.147720 (+ 1060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:08:49.147730 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:08:49.148947 (+ 1217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:08:49.148959 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:08:49.149755 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:08:49.149764 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:08:49.150101 (+ 337us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:08:49.150917 (+ 816us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:08:49.150955 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:08:49.154179 (+ 3224us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:08:49.158154 (+ 3975us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:08:49.158167 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:08:49.158191 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:08:49.158763 (+ 572us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:08:49.159188 (+ 425us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:08:49.159195 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:08:49.159199 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:08:49.159441 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:08:49.160024 (+ 583us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:08:49.160034 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:08:49.160439 (+ 405us) client_negotiation.cc:769] Sending connection context
0923 20:08:49.160725 (+ 286us) client_negotiation.cc:240] Negotiation successful
0923 20:08:49.161332 (+ 607us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1040,"thread_start_us":844,"threads_started":1}
I20260923 20:08:49.164229 2738 heartbeater.cc:347] Connected to a master server at 127.1.170.190:40377
I20260923 20:08:49.164577 2738 heartbeater.cc:464] Registering TS with master...
I20260923 20:08:49.165338 2738 heartbeater.cc:511] Master 127.1.170.190:40377 requested a full tablet report, sending...
I20260923 20:08:49.167716 2282 ts_manager.cc:194] Registered new tserver with Master: d714201f8d2d4b989623b62e603c6d1e (127.1.170.131:39233)
I20260923 20:08:49.168908 2282 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.170.131@KRBTEST.COM'} at 127.1.170.131:46833
I20260923 20:08:49.169943 1706 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260923 20:08:49.172194 2471 heartbeater.cc:503] Master 127.1.170.190:40377 was elected leader, sending a full tablet report...
I20260923 20:08:49.195871 2473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:08:49.177399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48376 (local address 127.1.170.190:40377)
0923 20:08:49.177629 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0923 20:08:49.177636 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:08:49.178569 (+ 933us) server_negotiation.cc:407] Connection header received
0923 20:08:49.178780 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:08:49.178787 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:08:49.178881 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:08:49.178982 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:08:49.181435 (+ 2453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:08:49.182262 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:08:49.184073 (+ 1811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:08:49.184473 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:08:49.184617 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:08:49.187985 (+ 3368us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:08:49.187991 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:08:49.187995 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:08:49.188029 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:08:49.191979 (+ 3950us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:08:49.192852 (+ 873us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:08:49.192859 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:08:49.192868 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:08:49.193085 (+ 217us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:08:49.193782 (+ 697us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:08:49.193788 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:08:49.193791 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:08:49.194334 (+ 543us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:08:49.194706 (+ 372us) server_negotiation.cc:1035] Waiting for connection context
0923 20:08:49.195122 (+ 416us) server_negotiation.cc:299] Negotiation successful
0923 20:08:49.195404 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260923 20:08:49.206789 2281 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48376:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 20:08:49.217470 2281 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 20:08:49.257493 2751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:08:49.237199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51158 (local address 127.1.170.130:45725)
0923 20:08:49.239314 (+ 2115us) server_negotiation.cc:206] Beginning negotiation
0923 20:08:49.239334 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0923 20:08:49.241066 (+ 1732us) server_negotiation.cc:407] Connection header received
0923 20:08:49.241179 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:08:49.241186 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:08:49.241588 (+ 402us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:08:49.242031 (+ 443us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:08:49.243854 (+ 1823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:08:49.246577 (+ 2723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:08:49.252649 (+ 6072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:08:49.255038 (+ 2389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:08:49.255294 (+ 256us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:08:49.255580 (+ 286us) server_negotiation.cc:1035] Waiting for connection context
0923 20:08:49.256633 (+ 1053us) server_negotiation.cc:299] Negotiation successful
0923 20:08:49.256897 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1812,"thread_start_us":1517,"threads_started":1}
I20260923 20:08:49.258942 2753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:08:49.236570 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.130:45725 (local address 127.0.0.1:51158)
0923 20:08:49.240435 (+ 3865us) negotiation.cc:107] Waiting for socket to connect
0923 20:08:49.240508 (+ 73us) client_negotiation.cc:174] Beginning negotiation
0923 20:08:49.240856 (+ 348us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:08:49.242483 (+ 1627us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:08:49.242494 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:08:49.242539 (+ 45us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:08:49.243446 (+ 907us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:08:49.243463 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:08:49.246823 (+ 3360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:08:49.246832 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:08:49.252439 (+ 5607us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:08:49.252454 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:08:49.255885 (+ 3431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:08:49.255902 (+ 17us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:08:49.256242 (+ 340us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:08:49.256303 (+ 61us) client_negotiation.cc:769] Sending connection context
0923 20:08:49.257771 (+ 1468us) client_negotiation.cc:240] Negotiation successful
0923 20:08:49.258160 (+ 389us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3518,"thread_start_us":11986,"threads_started":1}
I20260923 20:08:49.262681 2755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:08:49.241901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39386 (local address 127.1.170.129:44597)
0923 20:08:49.248128 (+ 6227us) server_negotiation.cc:206] Beginning negotiation
0923 20:08:49.248141 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0923 20:08:49.251413 (+ 3272us) server_negotiation.cc:407] Connection header received
0923 20:08:49.252344 (+ 931us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:08:49.252365 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:08:49.252750 (+ 385us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:08:49.253128 (+ 378us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:08:49.254185 (+ 1057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:08:49.256010 (+ 1825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:08:49.259382 (+ 3372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:08:49.260825 (+ 1443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:08:49.260967 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:08:49.261320 (+ 353us) server_negotiation.cc:1035] Waiting for connection context
0923 20:08:49.261831 (+ 511us) server_negotiation.cc:299] Negotiation successful
0923 20:08:49.262050 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5938,"thread_start_us":5702,"threads_started":1}
I20260923 20:08:49.263221 2752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:08:49.238416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54916 (local address 127.1.170.131:39233)
0923 20:08:49.240164 (+ 1748us) server_negotiation.cc:206] Beginning negotiation
0923 20:08:49.240172 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 20:08:49.250563 (+ 10391us) server_negotiation.cc:407] Connection header received
0923 20:08:49.250845 (+ 282us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:08:49.250853 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:08:49.251201 (+ 348us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:08:49.251568 (+ 367us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:08:49.252924 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:08:49.255222 (+ 2298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:08:49.259357 (+ 4135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:08:49.261398 (+ 2041us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:08:49.261577 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:08:49.261864 (+ 287us) server_negotiation.cc:1035] Waiting for connection context
0923 20:08:49.262228 (+ 364us) server_negotiation.cc:299] Negotiation successful
0923 20:08:49.262487 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1463,"thread_start_us":1103,"threads_started":1}
I20260923 20:08:49.263046 2750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:08:49.237695 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.131:39233 (local address 127.0.0.1:54916)
0923 20:08:49.250304 (+ 12609us) negotiation.cc:107] Waiting for socket to connect
0923 20:08:49.250338 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0923 20:08:49.250646 (+ 308us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:08:49.251543 (+ 897us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:08:49.251547 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:08:49.251570 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:08:49.252660 (+ 1090us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:08:49.252674 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:08:49.255422 (+ 2748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:08:49.255427 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:08:49.259145 (+ 3718us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:08:49.259161 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:08:49.261718 (+ 2557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:08:49.261727 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:08:49.261983 (+ 256us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:08:49.262024 (+ 41us) client_negotiation.cc:769] Sending connection context
0923 20:08:49.262147 (+ 123us) client_negotiation.cc:240] Negotiation successful
0923 20:08:49.262320 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12404,"mutex_wait_us":7,"thread_start_us":2148,"threads_started":1}
I20260923 20:08:49.264169 2754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:08:49.241225 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.129:44597 (local address 127.0.0.1:39386)
0923 20:08:49.251100 (+ 9875us) negotiation.cc:107] Waiting for socket to connect
0923 20:08:49.251140 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0923 20:08:49.252050 (+ 910us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:08:49.253109 (+ 1059us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:08:49.253113 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:08:49.253145 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:08:49.253959 (+ 814us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:08:49.253972 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:08:49.256195 (+ 2223us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:08:49.256198 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:08:49.259156 (+ 2958us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:08:49.259172 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:08:49.261302 (+ 2130us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:08:49.261309 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:08:49.261564 (+ 255us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:08:49.261604 (+ 40us) client_negotiation.cc:769] Sending connection context
0923 20:08:49.263278 (+ 1674us) client_negotiation.cc:240] Negotiation successful
0923 20:08:49.263448 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9077,"mutex_wait_us":8,"thread_start_us":7945,"threads_started":1}
I20260923 20:08:49.265945 2540 tablet_service.cc:1467] Processing CreateTablet for tablet 2a23072d69144075ae4d3afd88ba8904 (DEFAULT_TABLE table=test-table [id=928c0523a9a747ea8f628e7a1558b5f6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 20:08:49.268759 2540 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2a23072d69144075ae4d3afd88ba8904. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:08:49.268605 2406 tablet_service.cc:1467] Processing CreateTablet for tablet 2a23072d69144075ae4d3afd88ba8904 (DEFAULT_TABLE table=test-table [id=928c0523a9a747ea8f628e7a1558b5f6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 20:08:49.268570 2673 tablet_service.cc:1467] Processing CreateTablet for tablet 2a23072d69144075ae4d3afd88ba8904 (DEFAULT_TABLE table=test-table [id=928c0523a9a747ea8f628e7a1558b5f6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 20:08:49.270782 2673 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2a23072d69144075ae4d3afd88ba8904. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:08:49.270831 2406 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2a23072d69144075ae4d3afd88ba8904. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:08:49.289012 2756 tablet_bootstrap.cc:492] T 2a23072d69144075ae4d3afd88ba8904 P 0791199741904b30aeb9576806fcc663: Bootstrap starting.
I20260923 20:08:49.293080 2758 tablet_bootstrap.cc:492] T 2a23072d69144075ae4d3afd88ba8904 P d714201f8d2d4b989623b62e603c6d1e: Bootstrap starting.
I20260923 20:08:49.293955 2757 tablet_bootstrap.cc:492] T 2a23072d69144075ae4d3afd88ba8904 P 4de9ea06b2704f4ca3fc1cd50ddd6e74: Bootstrap starting.
I20260923 20:08:49.297344 2756 tablet_bootstrap.cc:654] T 2a23072d69144075ae4d3afd88ba8904 P 0791199741904b30aeb9576806fcc663: Neither blocks nor log segments found. Creating new log.
I20260923 20:08:49.299153 2757 tablet_bootstrap.cc:654] T 2a23072d69144075ae4d3afd88ba8904 P 4de9ea06b2704f4ca3fc1cd50ddd6e74: Neither blocks nor log segments found. Creating new log.
I20260923 20:08:49.299094 2758 tablet_bootstrap.cc:654] T 2a23072d69144075ae4d3afd88ba8904 P d714201f8d2d4b989623b62e603c6d1e: Neither blocks nor log segments found. Creating new log.
I20260923 20:08:49.299867 2756 log.cc:824] T 2a23072d69144075ae4d3afd88ba8904 P 0791199741904b30aeb9576806fcc663: Log is configured to *not* fsync() on all Append() calls
I20260923 20:08:49.300800 2757 log.cc:824] T 2a23072d69144075ae4d3afd88ba8904 P 4de9ea06b2704f4ca3fc1cd50ddd6e74: Log is configured to *not* fsync() on all Append() calls
I20260923 20:08:49.301530 2758 log.cc:824] T 2a23072d69144075ae4d3afd88ba8904 P d714201f8d2d4b989623b62e603c6d1e: Log is configured to *not* fsync() on all Append() calls
I20260923 20:08:49.304730 2756 tablet_bootstrap.cc:492] T 2a23072d69144075ae4d3afd88ba8904 P 0791199741904b30aeb9576806fcc663: No bootstrap required, opened a new log
I20260923 20:08:49.305209 2756 ts_tablet_manager.cc:1397] T 2a23072d69144075ae4d3afd88ba8904 P 0791199741904b30aeb9576806fcc663: Time spent bootstrapping tablet: real 0.016s user 0.000s sys 0.014s
I20260923 20:08:49.305927 2758 tablet_bootstrap.cc:492] T 2a23072d69144075ae4d3afd88ba8904 P d714201f8d2d4b989623b62e603c6d1e: No bootstrap required, opened a new log
I20260923 20:08:49.306284 2758 ts_tablet_manager.cc:1397] T 2a23072d69144075ae4d3afd88ba8904 P d714201f8d2d4b989623b62e603c6d1e: Time spent bootstrapping tablet: real 0.014s user 0.007s sys 0.006s
I20260923 20:08:49.306396 2757 tablet_bootstrap.cc:492] T 2a23072d69144075ae4d3afd88ba8904 P 4de9ea06b2704f4ca3fc1cd50ddd6e74: No bootstrap required, opened a new log
I20260923 20:08:49.306806 2757 ts_tablet_manager.cc:1397] T 2a23072d69144075ae4d3afd88ba8904 P 4de9ea06b2704f4ca3fc1cd50ddd6e74: Time spent bootstrapping tablet: real 0.013s user 0.002s sys 0.009s
I20260923 20:08:49.319290 2758 raft_consensus.cc:348] T 2a23072d69144075ae4d3afd88ba8904 P d714201f8d2d4b989623b62e603c6d1e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0791199741904b30aeb9576806fcc663" member_type: VOTER last_known_addr { host: "127.1.170.130" port: 45725 } } peers { permanent_uuid: "d714201f8d2d4b989623b62e603c6d1e" member_type: VOTER last_known_addr { host: "127.1.170.131" port: 39233 } } peers { permanent_uuid: "4de9ea06b2704f4ca3fc1cd50ddd6e74" member_type: VOTER last_known_addr { host: "127.1.170.129" port: 44597 } }
I20260923 20:08:49.320446 2758 raft_consensus.cc:374] T 2a23072d69144075ae4d3afd88ba8904 P d714201f8d2d4b989623b62e603c6d1e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:08:49.320945 2758 raft_consensus.cc:729] T 2a23072d69144075ae4d3afd88ba8904 P d714201f8d2d4b989623b62e603c6d1e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d714201f8d2d4b989623b62e603c6d1e, State: Initialized, Role: FOLLOWER
I20260923 20:08:49.322217 2758 consensus_queue.cc:260] T 2a23072d69144075ae4d3afd88ba8904 P d714201f8d2d4b989623b62e603c6d1e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0791199741904b30aeb9576806fcc663" member_type: VOTER last_known_addr { host: "127.1.170.130" port: 45725 } } peers { permanent_uuid: "d714201f8d2d4b989623b62e603c6d1e" member_type: VOTER last_known_addr { host: "127.1.170.131" port: 39233 } } peers { permanent_uuid: "4de9ea06b2704f4ca3fc1cd50ddd6e74" member_type: VOTER last_known_addr { host: "127.1.170.129" port: 44597 } }
I20260923 20:08:49.324249 2738 heartbeater.cc:503] Master 127.1.170.190:40377 was elected leader, sending a full tablet report...
I20260923 20:08:49.324106 2756 raft_consensus.cc:348] T 2a23072d69144075ae4d3afd88ba8904 P 0791199741904b30aeb9576806fcc663 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0791199741904b30aeb9576806fcc663" member_type: VOTER last_known_addr { host: "127.1.170.130" port: 45725 } } peers { permanent_uuid: "d714201f8d2d4b989623b62e603c6d1e" member_type: VOTER last_known_addr { host: "127.1.170.131" port: 39233 } } peers { permanent_uuid: "4de9ea06b2704f4ca3fc1cd50ddd6e74" member_type: VOTER last_known_addr { host: "127.1.170.129" port: 44597 } }
I20260923 20:08:49.324779 2756 raft_consensus.cc:374] T 2a23072d69144075ae4d3afd88ba8904 P 0791199741904b30aeb9576806fcc663 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:08:49.324946 2756 raft_consensus.cc:729] T 2a23072d69144075ae4d3afd88ba8904 P 0791199741904b30aeb9576806fcc663 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0791199741904b30aeb9576806fcc663, State: Initialized, Role: FOLLOWER
I20260923 20:08:49.325286 2758 ts_tablet_manager.cc:1428] T 2a23072d69144075ae4d3afd88ba8904 P d714201f8d2d4b989623b62e603c6d1e: Time spent starting tablet: real 0.019s user 0.015s sys 0.002s
I20260923 20:08:49.325031 2757 raft_consensus.cc:348] T 2a23072d69144075ae4d3afd88ba8904 P 4de9ea06b2704f4ca3fc1cd50ddd6e74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0791199741904b30aeb9576806fcc663" member_type: VOTER last_known_addr { host: "127.1.170.130" port: 45725 } } peers { permanent_uuid: "d714201f8d2d4b989623b62e603c6d1e" member_type: VOTER last_known_addr { host: "127.1.170.131" port: 39233 } } peers { permanent_uuid: "4de9ea06b2704f4ca3fc1cd50ddd6e74" member_type: VOTER last_known_addr { host: "127.1.170.129" port: 44597 } }
I20260923 20:08:49.325805 2757 raft_consensus.cc:374] T 2a23072d69144075ae4d3afd88ba8904 P 4de9ea06b2704f4ca3fc1cd50ddd6e74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:08:49.326009 2757 raft_consensus.cc:729] T 2a23072d69144075ae4d3afd88ba8904 P 4de9ea06b2704f4ca3fc1cd50ddd6e74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4de9ea06b2704f4ca3fc1cd50ddd6e74, State: Initialized, Role: FOLLOWER
I20260923 20:08:49.326048 2756 consensus_queue.cc:260] T 2a23072d69144075ae4d3afd88ba8904 P 0791199741904b30aeb9576806fcc663 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0791199741904b30aeb9576806fcc663" member_type: VOTER last_known_addr { host: "127.1.170.130" port: 45725 } } peers { permanent_uuid: "d714201f8d2d4b989623b62e603c6d1e" member_type: VOTER last_known_addr { host: "127.1.170.131" port: 39233 } } peers { permanent_uuid: "4de9ea06b2704f4ca3fc1cd50ddd6e74" member_type: VOTER last_known_addr { host: "127.1.170.129" port: 44597 } }
I20260923 20:08:49.326874 2757 consensus_queue.cc:260] T 2a23072d69144075ae4d3afd88ba8904 P 4de9ea06b2704f4ca3fc1cd50ddd6e74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0791199741904b30aeb9576806fcc663" member_type: VOTER last_known_addr { host: "127.1.170.130" port: 45725 } } peers { permanent_uuid: "d714201f8d2d4b989623b62e603c6d1e" member_type: VOTER last_known_addr { host: "127.1.170.131" port: 39233 } } peers { permanent_uuid: "4de9ea06b2704f4ca3fc1cd50ddd6e74" member_type: VOTER last_known_addr { host: "127.1.170.129" port: 44597 } }
I20260923 20:08:49.329770 2757 ts_tablet_manager.cc:1428] T 2a23072d69144075ae4d3afd88ba8904 P 4de9ea06b2704f4ca3fc1cd50ddd6e74: Time spent starting tablet: real 0.023s user 0.015s sys 0.006s
I20260923 20:08:49.331233 2605 heartbeater.cc:503] Master 127.1.170.190:40377 was elected leader, sending a full tablet report...
I20260923 20:08:49.332533 2756 ts_tablet_manager.cc:1428] T 2a23072d69144075ae4d3afd88ba8904 P 0791199741904b30aeb9576806fcc663: Time spent starting tablet: real 0.027s user 0.018s sys 0.005s
W20260923 20:08:49.381668 2606 tablet.cc:2369] T 2a23072d69144075ae4d3afd88ba8904 P 0791199741904b30aeb9576806fcc663: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260923 20:08:49.390021 2739 tablet.cc:2369] T 2a23072d69144075ae4d3afd88ba8904 P d714201f8d2d4b989623b62e603c6d1e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260923 20:08:49.390260 2472 tablet.cc:2369] T 2a23072d69144075ae4d3afd88ba8904 P 4de9ea06b2704f4ca3fc1cd50ddd6e74: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 20:08:49.442830 2762 raft_consensus.cc:482] T 2a23072d69144075ae4d3afd88ba8904 P d714201f8d2d4b989623b62e603c6d1e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 20:08:49.443311 2762 raft_consensus.cc:504] T 2a23072d69144075ae4d3afd88ba8904 P d714201f8d2d4b989623b62e603c6d1e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0791199741904b30aeb9576806fcc663" member_type: VOTER last_known_addr { host: "127.1.170.130" port: 45725 } } peers { permanent_uuid: "d714201f8d2d4b989623b62e603c6d1e" member_type: VOTER last_known_addr { host: "127.1.170.131" port: 39233 } } peers { permanent_uuid: "4de9ea06b2704f4ca3fc1cd50ddd6e74" member_type: VOTER last_known_addr { host: "127.1.170.129" port: 44597 } }
I20260923 20:08:49.445662 2762 leader_election.cc:288] T 2a23072d69144075ae4d3afd88ba8904 P d714201f8d2d4b989623b62e603c6d1e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0791199741904b30aeb9576806fcc663 (127.1.170.130:45725), 4de9ea06b2704f4ca3fc1cd50ddd6e74 (127.1.170.129:44597)
I20260923 20:08:49.455879 2740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:08:49.446421 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.130:45725 (local address 127.1.170.131:33701)
0923 20:08:49.446728 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0923 20:08:49.446752 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0923 20:08:49.446943 (+ 191us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:08:49.447497 (+ 554us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:08:49.447510 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:08:49.447563 (+ 53us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:08:49.448054 (+ 491us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:08:49.448079 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:08:49.450499 (+ 2420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:08:49.450505 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:08:49.453251 (+ 2746us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:08:49.453270 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:08:49.455033 (+ 1763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:08:49.455037 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:08:49.455248 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:08:49.455276 (+ 28us) client_negotiation.cc:769] Sending connection context
0923 20:08:49.455358 (+ 82us) client_negotiation.cc:240] Negotiation successful
0923 20:08:49.455461 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"mutex_wait_us":31}
I20260923 20:08:49.455964 2751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:08:49.446106 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.170.131:33701 (local address 127.1.170.130:45725)
0923 20:08:49.446507 (+ 401us) server_negotiation.cc:206] Beginning negotiation
0923 20:08:49.446524 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0923 20:08:49.446909 (+ 385us) server_negotiation.cc:407] Connection header received
0923 20:08:49.447146 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:08:49.447154 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:08:49.447266 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:08:49.447402 (+ 136us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:08:49.448263 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:08:49.449773 (+ 1510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:08:49.453508 (+ 3735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:08:49.454855 (+ 1347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:08:49.455053 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:08:49.455142 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0923 20:08:49.455411 (+ 269us) server_negotiation.cc:299] Negotiation successful
0923 20:08:49.455569 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":145}
I20260923 20:08:49.457528 2755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:08:49.446435 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.170.131:55693 (local address 127.1.170.129:44597)
0923 20:08:49.446761 (+ 326us) server_negotiation.cc:206] Beginning negotiation
0923 20:08:49.446767 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:08:49.448536 (+ 1769us) server_negotiation.cc:407] Connection header received
0923 20:08:49.448634 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:08:49.448639 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:08:49.448731 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:08:49.448850 (+ 119us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:08:49.450015 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:08:49.451404 (+ 1389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:08:49.454708 (+ 3304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:08:49.456357 (+ 1649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:08:49.456607 (+ 250us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:08:49.456697 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0923 20:08:49.457180 (+ 483us) server_negotiation.cc:299] Negotiation successful
0923 20:08:49.457283 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138}
I20260923 20:08:49.457422 2560 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a23072d69144075ae4d3afd88ba8904" candidate_uuid: "d714201f8d2d4b989623b62e603c6d1e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0791199741904b30aeb9576806fcc663" is_pre_election: true
I20260923 20:08:49.458140 2765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:08:49.446732 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.129:44597 (local address 127.1.170.131:55693)
0923 20:08:49.448290 (+ 1558us) negotiation.cc:107] Waiting for socket to connect
0923 20:08:49.448321 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0923 20:08:49.448438 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:08:49.448886 (+ 448us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:08:49.448889 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:08:49.448908 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:08:49.449760 (+ 852us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:08:49.449773 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:08:49.451602 (+ 1829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:08:49.451608 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:08:49.454454 (+ 2846us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:08:49.454469 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:08:49.456654 (+ 2185us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:08:49.456664 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:08:49.456952 (+ 288us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:08:49.457005 (+ 53us) client_negotiation.cc:769] Sending connection context
0923 20:08:49.457121 (+ 116us) client_negotiation.cc:240] Negotiation successful
0923 20:08:49.457320 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1395,"mutex_wait_us":1,"thread_start_us":1004,"threads_started":1}
I20260923 20:08:49.458668 2560 raft_consensus.cc:2454] T 2a23072d69144075ae4d3afd88ba8904 P 0791199741904b30aeb9576806fcc663 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d714201f8d2d4b989623b62e603c6d1e in term 0.
I20260923 20:08:49.459462 2426 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a23072d69144075ae4d3afd88ba8904" candidate_uuid: "d714201f8d2d4b989623b62e603c6d1e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4de9ea06b2704f4ca3fc1cd50ddd6e74" is_pre_election: true
I20260923 20:08:49.459668 2626 leader_election.cc:302] T 2a23072d69144075ae4d3afd88ba8904 P d714201f8d2d4b989623b62e603c6d1e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0791199741904b30aeb9576806fcc663, d714201f8d2d4b989623b62e603c6d1e; no voters:
I20260923 20:08:49.460175 2426 raft_consensus.cc:2454] T 2a23072d69144075ae4d3afd88ba8904 P 4de9ea06b2704f4ca3fc1cd50ddd6e74 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d714201f8d2d4b989623b62e603c6d1e in term 0.
I20260923 20:08:49.460563 2762 raft_consensus.cc:2781] T 2a23072d69144075ae4d3afd88ba8904 P d714201f8d2d4b989623b62e603c6d1e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 20:08:49.460847 2762 raft_consensus.cc:482] T 2a23072d69144075ae4d3afd88ba8904 P d714201f8d2d4b989623b62e603c6d1e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 20:08:49.461078 2762 raft_consensus.cc:3037] T 2a23072d69144075ae4d3afd88ba8904 P d714201f8d2d4b989623b62e603c6d1e [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:08:49.464495 2762 raft_consensus.cc:504] T 2a23072d69144075ae4d3afd88ba8904 P d714201f8d2d4b989623b62e603c6d1e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0791199741904b30aeb9576806fcc663" member_type: VOTER last_known_addr { host: "127.1.170.130" port: 45725 } } peers { permanent_uuid: "d714201f8d2d4b989623b62e603c6d1e" member_type: VOTER last_known_addr { host: "127.1.170.131" port: 39233 } } peers { permanent_uuid: "4de9ea06b2704f4ca3fc1cd50ddd6e74" member_type: VOTER last_known_addr { host: "127.1.170.129" port: 44597 } }
I20260923 20:08:49.465287 2762 leader_election.cc:288] T 2a23072d69144075ae4d3afd88ba8904 P d714201f8d2d4b989623b62e603c6d1e [CANDIDATE]: Term 1 election: Requested vote from peers 0791199741904b30aeb9576806fcc663 (127.1.170.130:45725), 4de9ea06b2704f4ca3fc1cd50ddd6e74 (127.1.170.129:44597)
I20260923 20:08:49.465910 2560 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a23072d69144075ae4d3afd88ba8904" candidate_uuid: "d714201f8d2d4b989623b62e603c6d1e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0791199741904b30aeb9576806fcc663"
I20260923 20:08:49.465955 2426 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a23072d69144075ae4d3afd88ba8904" candidate_uuid: "d714201f8d2d4b989623b62e603c6d1e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4de9ea06b2704f4ca3fc1cd50ddd6e74"
I20260923 20:08:49.466197 2560 raft_consensus.cc:3037] T 2a23072d69144075ae4d3afd88ba8904 P 0791199741904b30aeb9576806fcc663 [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:08:49.466223 2426 raft_consensus.cc:3037] T 2a23072d69144075ae4d3afd88ba8904 P 4de9ea06b2704f4ca3fc1cd50ddd6e74 [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:08:49.469455 2560 raft_consensus.cc:2454] T 2a23072d69144075ae4d3afd88ba8904 P 0791199741904b30aeb9576806fcc663 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d714201f8d2d4b989623b62e603c6d1e in term 1.
I20260923 20:08:49.469568 2426 raft_consensus.cc:2454] T 2a23072d69144075ae4d3afd88ba8904 P 4de9ea06b2704f4ca3fc1cd50ddd6e74 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d714201f8d2d4b989623b62e603c6d1e in term 1.
I20260923 20:08:49.470064 2626 leader_election.cc:302] T 2a23072d69144075ae4d3afd88ba8904 P d714201f8d2d4b989623b62e603c6d1e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0791199741904b30aeb9576806fcc663, d714201f8d2d4b989623b62e603c6d1e; no voters:
I20260923 20:08:49.470470 2762 raft_consensus.cc:2781] T 2a23072d69144075ae4d3afd88ba8904 P d714201f8d2d4b989623b62e603c6d1e [term 1 FOLLOWER]: Leader election won for term 1
I20260923 20:08:49.471182 2762 raft_consensus.cc:686] T 2a23072d69144075ae4d3afd88ba8904 P d714201f8d2d4b989623b62e603c6d1e [term 1 LEADER]: Becoming Leader. State: Replica: d714201f8d2d4b989623b62e603c6d1e, State: Running, Role: LEADER
I20260923 20:08:49.471952 2762 consensus_queue.cc:237] T 2a23072d69144075ae4d3afd88ba8904 P d714201f8d2d4b989623b62e603c6d1e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0791199741904b30aeb9576806fcc663" member_type: VOTER last_known_addr { host: "127.1.170.130" port: 45725 } } peers { permanent_uuid: "d714201f8d2d4b989623b62e603c6d1e" member_type: VOTER last_known_addr { host: "127.1.170.131" port: 39233 } } peers { permanent_uuid: "4de9ea06b2704f4ca3fc1cd50ddd6e74" member_type: VOTER last_known_addr { host: "127.1.170.129" port: 44597 } }
I20260923 20:08:49.478436 2280 catalog_manager.cc:5495] T 2a23072d69144075ae4d3afd88ba8904 P d714201f8d2d4b989623b62e603c6d1e reported cstate change: term changed from 0 to 1, leader changed from <none> to d714201f8d2d4b989623b62e603c6d1e (127.1.170.131). New cstate: current_term: 1 leader_uuid: "d714201f8d2d4b989623b62e603c6d1e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0791199741904b30aeb9576806fcc663" member_type: VOTER last_known_addr { host: "127.1.170.130" port: 45725 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d714201f8d2d4b989623b62e603c6d1e" member_type: VOTER last_known_addr { host: "127.1.170.131" port: 39233 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4de9ea06b2704f4ca3fc1cd50ddd6e74" member_type: VOTER last_known_addr { host: "127.1.170.129" port: 44597 } health_report { overall_health: UNKNOWN } } }
Sep 23 20:08:49 dist-test-slave-th4g krb5kdc[2237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194127, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.170.129@KRBTEST.COM
I20260923 20:08:49.525195 2755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:08:49.501110 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39402 (local address 127.1.170.129:44597)
0923 20:08:49.501280 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0923 20:08:49.501285 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 20:08:49.502564 (+ 1279us) server_negotiation.cc:407] Connection header received
0923 20:08:49.502737 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:08:49.502741 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:08:49.502807 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:08:49.502895 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:08:49.505330 (+ 2435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:08:49.506425 (+ 1095us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:08:49.508489 (+ 2064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:08:49.509144 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:08:49.509273 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:08:49.515995 (+ 6722us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:08:49.516012 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:08:49.516029 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:08:49.516090 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:08:49.520629 (+ 4539us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:08:49.521784 (+ 1155us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:08:49.521792 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:08:49.521794 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:08:49.522103 (+ 309us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:08:49.522680 (+ 577us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:08:49.522686 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:08:49.522689 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:08:49.523463 (+ 774us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:08:49.523937 (+ 474us) server_negotiation.cc:1035] Waiting for connection context
0923 20:08:49.524444 (+ 507us) server_negotiation.cc:299] Negotiation successful
0923 20:08:49.524633 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
Sep 23 20:08:49 dist-test-slave-th4g krb5kdc[2237](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194129, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 23 20:08:49 dist-test-slave-th4g krb5kdc[2237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194129, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.1.170.129@KRBTEST.COM
I20260923 20:08:49.587519 2755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:08:49.564925 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39416 (local address 127.1.170.129:44597)
0923 20:08:49.565282 (+ 357us) server_negotiation.cc:206] Beginning negotiation
0923 20:08:49.565299 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0923 20:08:49.565859 (+ 560us) server_negotiation.cc:407] Connection header received
0923 20:08:49.566146 (+ 287us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:08:49.566150 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:08:49.566218 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:08:49.566333 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:08:49.568577 (+ 2244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:08:49.569621 (+ 1044us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:08:49.571678 (+ 2057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:08:49.572709 (+ 1031us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:08:49.572891 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:08:49.578662 (+ 5771us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:08:49.578669 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:08:49.578673 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:08:49.578724 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:08:49.582772 (+ 4048us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:08:49.584239 (+ 1467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:08:49.584245 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:08:49.584248 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:08:49.584862 (+ 614us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:08:49.585532 (+ 670us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:08:49.585537 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:08:49.585539 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:08:49.586036 (+ 497us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:08:49.586468 (+ 432us) server_negotiation.cc:1035] Waiting for connection context
0923 20:08:49.586800 (+ 332us) server_negotiation.cc:299] Negotiation successful
0923 20:08:49.587024 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120}
Sep 23 20:08:49 dist-test-slave-th4g krb5kdc[2237](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194129, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 23 20:08:49 dist-test-slave-th4g krb5kdc[2237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194129, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.170.129@KRBTEST.COM
I20260923 20:08:49.646534 2755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:08:49.623035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39420 (local address 127.1.170.129:44597)
0923 20:08:49.623388 (+ 353us) server_negotiation.cc:206] Beginning negotiation
0923 20:08:49.623396 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 20:08:49.623624 (+ 228us) server_negotiation.cc:407] Connection header received
0923 20:08:49.623860 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:08:49.623864 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:08:49.623925 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:08:49.624037 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:08:49.626104 (+ 2067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:08:49.626962 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:08:49.629365 (+ 2403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:08:49.630144 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:08:49.630293 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:08:49.638062 (+ 7769us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:08:49.638072 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:08:49.638079 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:08:49.638143 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:08:49.642538 (+ 4395us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:08:49.643730 (+ 1192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:08:49.643738 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:08:49.643746 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:08:49.643945 (+ 199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:08:49.644375 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:08:49.644379 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:08:49.644381 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:08:49.644960 (+ 579us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:08:49.645216 (+ 256us) server_negotiation.cc:1035] Waiting for connection context
0923 20:08:49.645535 (+ 319us) server_negotiation.cc:299] Negotiation successful
0923 20:08:49.645847 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
W20260923 20:08:49.648964 2386 tablet_service.cc:2908] Rejecting scan request for tablet 2a23072d69144075ae4d3afd88ba8904: Uninitialized: clean time has not yet been initialized
I20260923 20:08:49.653685 1706 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskqUuku6/build/asan/bin/kudu pid 2343
I20260923 20:08:49.668540 2755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:08:49.654778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39426 (local address 127.1.170.129:44597)
0923 20:08:49.654959 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0923 20:08:49.654963 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 20:08:49.655339 (+ 376us) server_negotiation.cc:407] Connection header received
0923 20:08:49.655442 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:08:49.655446 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:08:49.655522 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:08:49.655610 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:08:49.656974 (+ 1364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:08:49.658089 (+ 1115us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:08:49.659413 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:08:49.659824 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:08:49.659931 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:08:49.662849 (+ 2918us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:08:49.662855 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:08:49.662858 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:08:49.662896 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:08:49.665766 (+ 2870us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:08:49.666581 (+ 815us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:08:49.666585 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:08:49.666588 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:08:49.666715 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:08:49.667076 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:08:49.667080 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:08:49.667083 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:08:49.667493 (+ 410us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:08:49.667704 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0923 20:08:49.667967 (+ 263us) server_negotiation.cc:299] Negotiation successful
0923 20:08:49.668199 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260923 20:08:49.669644 2466 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:08:49.762490 1706 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqUuku6/build/asan/bin/kudu with pid 2343
I20260923 20:08:49.779714 1706 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskqUuku6/build/asan/bin/kudu pid 2477
Sep 23 20:08:49 dist-test-slave-th4g krb5kdc[2237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194129, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.170.130@KRBTEST.COM
I20260923 20:08:49.796197 2751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:08:49.780711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51160 (local address 127.1.170.130:45725)
0923 20:08:49.780932 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0923 20:08:49.780939 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:08:49.780961 (+ 22us) server_negotiation.cc:407] Connection header received
0923 20:08:49.781037 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:08:49.781042 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:08:49.781113 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:08:49.781223 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:08:49.782871 (+ 1648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:08:49.783689 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:08:49.784973 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:08:49.785371 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:08:49.785510 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:08:49.790038 (+ 4528us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:08:49.790054 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:08:49.790072 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:08:49.790116 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:08:49.793074 (+ 2958us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:08:49.794009 (+ 935us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:08:49.794018 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:08:49.794020 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:08:49.794163 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:08:49.794602 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:08:49.794607 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:08:49.794613 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:08:49.795000 (+ 387us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:08:49.795337 (+ 337us) server_negotiation.cc:1035] Waiting for connection context
0923 20:08:49.795617 (+ 280us) server_negotiation.cc:299] Negotiation successful
0923 20:08:49.795859 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96}
I20260923 20:08:49.797046 2600 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:08:49.888193 1706 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqUuku6/build/asan/bin/kudu with pid 2477
I20260923 20:08:49.905076 1706 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskqUuku6/build/asan/bin/kudu pid 2610
Sep 23 20:08:49 dist-test-slave-th4g krb5kdc[2237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194129, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.170.131@KRBTEST.COM
I20260923 20:08:49.921494 2781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:08:49.905919 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54922 (local address 127.1.170.131:39233)
0923 20:08:49.906409 (+ 490us) server_negotiation.cc:206] Beginning negotiation
0923 20:08:49.906414 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 20:08:49.906431 (+ 17us) server_negotiation.cc:407] Connection header received
0923 20:08:49.906483 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:08:49.906487 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:08:49.906541 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:08:49.906634 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:08:49.907893 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:08:49.908586 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:08:49.910048 (+ 1462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:08:49.910454 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:08:49.910688 (+ 234us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:08:49.915182 (+ 4494us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:08:49.915200 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:08:49.915217 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:08:49.915256 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:08:49.918247 (+ 2991us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:08:49.919069 (+ 822us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:08:49.919076 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:08:49.919079 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:08:49.919266 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:08:49.919769 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:08:49.919775 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:08:49.919780 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:08:49.920140 (+ 360us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:08:49.920437 (+ 297us) server_negotiation.cc:1035] Waiting for connection context
0923 20:08:49.920716 (+ 279us) server_negotiation.cc:299] Negotiation successful
0923 20:08:49.920978 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":398,"thread_start_us":281,"threads_started":1}
I20260923 20:08:49.922200 2733 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:08:50.022644 1706 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqUuku6/build/asan/bin/kudu with pid 2610
I20260923 20:08:50.036231 1706 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskqUuku6/build/asan/bin/kudu pid 2250
I20260923 20:08:50.038017 2312 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:08:50.149818 1706 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqUuku6/build/asan/bin/kudu with pid 2250
2026-09-23T20:08:50Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (4349 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790194121577027-1706-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 20:08:50 dist-test-slave-th4g krb5kdc[2786](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 20:08:50 dist-test-slave-th4g krb5kdc[2786](info): set up 2 sockets
Sep 23 20:08:50 dist-test-slave-th4g krb5kdc[2786](info): commencing operation
krb5kdc: starting...
W20260923 20:08:51.278240 1706 mini_kdc.cc:121] Time spent starting KDC: real 1.085s user 0.002s sys 0.014s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790194121577027-1706-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790194121577027-1706-0/krb5kdc/test-user.keytab.
Sep 23 20:08:51 dist-test-slave-th4g krb5kdc[2786](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194131, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T20:08:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T20:08:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.170.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.170.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.170.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790194121577027-1706-0/krb5kdc/kudu_127.1.170.190.keytab.
Entry for principal kudu/127.1.170.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790194121577027-1706-0/krb5kdc/kudu_127.1.170.190.keytab.
I20260923 20:08:51.467422 1706 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqUuku6/build/asan/bin/kudu
/tmp/dist-test-taskqUuku6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790194121577027-1706-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790194121577027-1706-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790194121577027-1706-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790194121577027-1706-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.170.190:39075
--webserver_interface=127.1.170.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790194121577027-1706-0/krb5kdc/kudu_127.1.170.190.keytab
--principal=kudu/127.1.170.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.170.148:41579
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.170.190:39075
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790194121577027-1706-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790194121577027-1706-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790194121577027-1706-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:08:51.784636 2799 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:08:51.784915 2799 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:08:51.785004 2799 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:08:51.792577 2799 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 20:08:51.792688 2799 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260923 20:08:51.792734 2799 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:08:51.792789 2799 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:08:51.792837 2799 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 20:08:51.792891 2799 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 20:08:51.803529 2799 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.170.148:41579
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790194121577027-1706-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790194121577027-1706-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.170.190:39075
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790194121577027-1706-0/krb5kdc/kudu_127.1.170.190.keytab
--principal=kudu/127.1.170.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.170.190:39075
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790194121577027-1706-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.170.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790194121577027-1706-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:08:51.804795 2799 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:08:51.806041 2799 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:08:51.817495 2805 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:08:51.817487 2804 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:08:51.817549 2807 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:08:51.818550 2799 server_base.cc:1034] running on GCE node
I20260923 20:08:51.819278 2799 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:08:51.820600 2799 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:08:51.821846 2799 hybrid_clock.cc:648] HybridClock initialized: now 1790194131821792 us; error 24 us; skew 500 ppm
I20260923 20:08:51.822235 2799 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:08:51 dist-test-slave-th4g krb5kdc[2786](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194131, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.170.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:08:51.825419 2799 init.cc:377] Logged in from keytab as kudu/127.1.170.190@KRBTEST.COM (short username kudu)
I20260923 20:08:51.826856 2799 webserver.cc:460] Webserver started at http://127.1.170.190:39639/ using document root <none> and password file <none>
I20260923 20:08:51.827497 2799 fs_manager.cc:359] Metadata directory not provided
I20260923 20:08:51.827579 2799 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:08:51.827828 2799 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:08:51.830701 2799 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790194121577027-1706-0/minicluster-data/master-0/data/instance:
uuid: "be44f549a3ba42f48a99b54e16576e8a"
format_stamp: "Formatted at 2026-09-23 20:08:51 on dist-test-slave-th4g"
server_key: "26c339f0583b00b59347494e19a8aebe"
server_key_iv: "823a5e654c561661440ba76212ee6182"
server_key_version: "encryptionkey@0"
I20260923 20:08:51.831419 2799 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790194121577027-1706-0/minicluster-data/master-0/wal/instance:
uuid: "be44f549a3ba42f48a99b54e16576e8a"
format_stamp: "Formatted at 2026-09-23 20:08:51 on dist-test-slave-th4g"
server_key: "26c339f0583b00b59347494e19a8aebe"
server_key_iv: "823a5e654c561661440ba76212ee6182"
server_key_version: "encryptionkey@0"
I20260923 20:08:51.836524 2799 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260923 20:08:51.839994 2814 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:08:51.841535 2799 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.001s
I20260923 20:08:51.841733 2799 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790194121577027-1706-0/minicluster-data/master-0/data,/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790194121577027-1706-0/minicluster-data/master-0/wal
uuid: "be44f549a3ba42f48a99b54e16576e8a"
format_stamp: "Formatted at 2026-09-23 20:08:51 on dist-test-slave-th4g"
server_key: "26c339f0583b00b59347494e19a8aebe"
server_key_iv: "823a5e654c561661440ba76212ee6182"
server_key_version: "encryptionkey@0"
I20260923 20:08:51.842028 2799 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790194121577027-1706-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790194121577027-1706-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790194121577027-1706-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:08:51.860249 2799 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:08:51.864413 2799 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:08:51.864688 2799 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:08:51.886861 2799 rpc_server.cc:307] RPC server started. Bound to: 127.1.170.190:39075
I20260923 20:08:51.886880 2875 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.170.190:39075 every 8 connection(s)
I20260923 20:08:51.888222 2799 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790194121577027-1706-0/minicluster-data/master-0/data/info.pb
I20260923 20:08:51.891841 2876 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:08:51.894647 1706 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqUuku6/build/asan/bin/kudu as pid 2799
I20260923 20:08:51.894903 1706 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790194121577027-1706-0/minicluster-data/master-0/wal/instance
I20260923 20:08:51.895879 1706 external_mini_cluster.cc:1428] Setting key 0ce913da72112a9fb96d636433828494
I20260923 20:08:51.904585 2876 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P be44f549a3ba42f48a99b54e16576e8a: Bootstrap starting.
I20260923 20:08:51.908304 2876 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P be44f549a3ba42f48a99b54e16576e8a: Neither blocks nor log segments found. Creating new log.
I20260923 20:08:51.909610 2876 log.cc:824] T 00000000000000000000000000000000 P be44f549a3ba42f48a99b54e16576e8a: Log is configured to *not* fsync() on all Append() calls
Sep 23 20:08:51 dist-test-slave-th4g krb5kdc[2786](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194131, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.170.190@KRBTEST.COM
I20260923 20:08:51.912846 2876 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P be44f549a3ba42f48a99b54e16576e8a: No bootstrap required, opened a new log
I20260923 20:08:51.919922 2879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:08:51.897944 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33280 (local address 127.1.170.190:39075)
0923 20:08:51.898753 (+ 809us) server_negotiation.cc:206] Beginning negotiation
0923 20:08:51.898765 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0923 20:08:51.898805 (+ 40us) server_negotiation.cc:407] Connection header received
0923 20:08:51.900423 (+ 1618us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:08:51.900445 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:08:51.900846 (+ 401us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:08:51.901169 (+ 323us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:08:51.902925 (+ 1756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:08:51.904187 (+ 1262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:08:51.905868 (+ 1681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:08:51.906640 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:08:51.906790 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:08:51.911399 (+ 4609us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:08:51.911409 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:08:51.911428 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:08:51.911466 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:08:51.915467 (+ 4001us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:08:51.916160 (+ 693us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:08:51.916170 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:08:51.916183 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:08:51.916372 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:08:51.916702 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:08:51.916706 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:08:51.916710 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:08:51.917394 (+ 684us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:08:51.917666 (+ 272us) server_negotiation.cc:1035] Waiting for connection context
0923 20:08:51.918089 (+ 423us) server_negotiation.cc:299] Negotiation successful
0923 20:08:51.918549 (+ 460us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":579,"thread_start_us":369,"threads_started":1}
I20260923 20:08:51.923153 2876 raft_consensus.cc:348] T 00000000000000000000000000000000 P be44f549a3ba42f48a99b54e16576e8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be44f549a3ba42f48a99b54e16576e8a" member_type: VOTER last_known_addr { host: "127.1.170.190" port: 39075 } }
I20260923 20:08:51.923488 2876 raft_consensus.cc:374] T 00000000000000000000000000000000 P be44f549a3ba42f48a99b54e16576e8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:08:51.923560 2876 raft_consensus.cc:729] T 00000000000000000000000000000000 P be44f549a3ba42f48a99b54e16576e8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: be44f549a3ba42f48a99b54e16576e8a, State: Initialized, Role: FOLLOWER
I20260923 20:08:51.924011 2876 consensus_queue.cc:260] T 00000000000000000000000000000000 P be44f549a3ba42f48a99b54e16576e8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be44f549a3ba42f48a99b54e16576e8a" member_type: VOTER last_known_addr { host: "127.1.170.190" port: 39075 } }
I20260923 20:08:51.924201 2876 raft_consensus.cc:388] T 00000000000000000000000000000000 P be44f549a3ba42f48a99b54e16576e8a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 20:08:51.924293 2876 raft_consensus.cc:482] T 00000000000000000000000000000000 P be44f549a3ba42f48a99b54e16576e8a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 20:08:51.924403 2876 raft_consensus.cc:3037] T 00000000000000000000000000000000 P be44f549a3ba42f48a99b54e16576e8a [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:08:51.926719 2876 raft_consensus.cc:504] T 00000000000000000000000000000000 P be44f549a3ba42f48a99b54e16576e8a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be44f549a3ba42f48a99b54e16576e8a" member_type: VOTER last_known_addr { host: "127.1.170.190" port: 39075 } }
I20260923 20:08:51.927196 2876 leader_election.cc:302] T 00000000000000000000000000000000 P be44f549a3ba42f48a99b54e16576e8a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: be44f549a3ba42f48a99b54e16576e8a; no voters:
I20260923 20:08:51.927783 2876 leader_election.cc:288] T 00000000000000000000000000000000 P be44f549a3ba42f48a99b54e16576e8a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 20:08:51.927954 2881 raft_consensus.cc:2781] T 00000000000000000000000000000000 P be44f549a3ba42f48a99b54e16576e8a [term 1 FOLLOWER]: Leader election won for term 1
I20260923 20:08:51.928714 2881 raft_consensus.cc:686] T 00000000000000000000000000000000 P be44f549a3ba42f48a99b54e16576e8a [term 1 LEADER]: Becoming Leader. State: Replica: be44f549a3ba42f48a99b54e16576e8a, State: Running, Role: LEADER
I20260923 20:08:51.929255 2881 consensus_queue.cc:237] T 00000000000000000000000000000000 P be44f549a3ba42f48a99b54e16576e8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be44f549a3ba42f48a99b54e16576e8a" member_type: VOTER last_known_addr { host: "127.1.170.190" port: 39075 } }
I20260923 20:08:51.929836 2876 sys_catalog.cc:564] T 00000000000000000000000000000000 P be44f549a3ba42f48a99b54e16576e8a [sys.catalog]: configured and running, proceeding with master startup.
I20260923 20:08:51.932495 2882 sys_catalog.cc:455] T 00000000000000000000000000000000 P be44f549a3ba42f48a99b54e16576e8a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "be44f549a3ba42f48a99b54e16576e8a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be44f549a3ba42f48a99b54e16576e8a" member_type: VOTER last_known_addr { host: "127.1.170.190" port: 39075 } } }
I20260923 20:08:51.932834 2882 sys_catalog.cc:458] T 00000000000000000000000000000000 P be44f549a3ba42f48a99b54e16576e8a [sys.catalog]: This master's current role is: LEADER
I20260923 20:08:51.934175 2883 sys_catalog.cc:455] T 00000000000000000000000000000000 P be44f549a3ba42f48a99b54e16576e8a [sys.catalog]: SysCatalogTable state changed. Reason: New leader be44f549a3ba42f48a99b54e16576e8a. Latest consensus state: current_term: 1 leader_uuid: "be44f549a3ba42f48a99b54e16576e8a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be44f549a3ba42f48a99b54e16576e8a" member_type: VOTER last_known_addr { host: "127.1.170.190" port: 39075 } } }
I20260923 20:08:51.934386 2888 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 20:08:51.934509 2883 sys_catalog.cc:458] T 00000000000000000000000000000000 P be44f549a3ba42f48a99b54e16576e8a [sys.catalog]: This master's current role is: LEADER
I20260923 20:08:51.939603 2888 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 20:08:51.948722 2888 catalog_manager.cc:1347] Generated new cluster ID: 6d41a5ec87bd4f058ed46cc8707b33ee
I20260923 20:08:51.948836 2888 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 20:08:51.974936 2888 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 20:08:51.976631 2888 catalog_manager.cc:1504] Loading token signing keys...
I20260923 20:08:51.987759 2888 catalog_manager.cc:5868] T 00000000000000000000000000000000 P be44f549a3ba42f48a99b54e16576e8a: Generated new TSK 0
I20260923 20:08:51.988596 2888 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.170.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.170.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.170.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790194121577027-1706-0/krb5kdc/kudu_127.1.170.129.keytab.
Entry for principal kudu/127.1.170.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790194121577027-1706-0/krb5kdc/kudu_127.1.170.129.keytab.
I20260923 20:08:52.047098 1706 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqUuku6/build/asan/bin/kudu
/tmp/dist-test-taskqUuku6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790194121577027-1706-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790194121577027-1706-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790194121577027-1706-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790194121577027-1706-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.170.129:0
--local_ip_for_outbound_sockets=127.1.170.129
--webserver_interface=127.1.170.129
--webserver_port=0
--tserver_master_addrs=127.1.170.190:39075
--keytab_file=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790194121577027-1706-0/krb5kdc/kudu_127.1.170.129.keytab
--principal=kudu/127.1.170.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.170.148:41579
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790194121577027-1706-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790194121577027-1706-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790194121577027-1706-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:08:52.401639 2902 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:08:52.401952 2902 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:08:52.402050 2902 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:08:52.409363 2902 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:08:52.409487 2902 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:08:52.409574 2902 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260923 20:08:52.409654 2902 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.170.129
I20260923 20:08:52.419613 2902 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.170.148:41579
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790194121577027-1706-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790194121577027-1706-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790194121577027-1706-0/krb5kdc/kudu_127.1.170.129.keytab
--principal=kudu/127.1.170.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.170.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790194121577027-1706-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.170.129
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.1.170.190:39075
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.170.129
--log_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790194121577027-1706-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:08:52.420826 2902 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:08:52.422097 2902 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:08:52.435106 2910 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:08:52.435972 2908 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:08:52.436208 2907 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:08:52.437100 2902 server_base.cc:1034] running on GCE node
I20260923 20:08:52.437758 2902 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:08:52.438813 2902 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:08:52.440078 2902 hybrid_clock.cc:648] HybridClock initialized: now 1790194132440017 us; error 45 us; skew 500 ppm
I20260923 20:08:52.440475 2902 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:08:52 dist-test-slave-th4g krb5kdc[2786](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194132, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.170.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:08:52.443959 2902 init.cc:377] Logged in from keytab as kudu/127.1.170.129@KRBTEST.COM (short username kudu)
I20260923 20:08:52.445760 2902 webserver.cc:460] Webserver started at http://127.1.170.129:45623/ using document root <none> and password file <none>
I20260923 20:08:52.446556 2902 fs_manager.cc:359] Metadata directory not provided
I20260923 20:08:52.446702 2902 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:08:52.447086 2902 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:08:52.450009 2902 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790194121577027-1706-0/minicluster-data/ts-0/data/instance:
uuid: "f1669496db8e44798ae7354b2d1f315b"
format_stamp: "Formatted at 2026-09-23 20:08:52 on dist-test-slave-th4g"
server_key: "9dcb7f46d2628d7d2d91335aea6d5219"
server_key_iv: "1c472c1bad968088b3957b735ccb942b"
server_key_version: "encryptionkey@0"
I20260923 20:08:52.450835 2902 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790194121577027-1706-0/minicluster-data/ts-0/wal/instance:
uuid: "f1669496db8e44798ae7354b2d1f315b"
format_stamp: "Formatted at 2026-09-23 20:08:52 on dist-test-slave-th4g"
server_key: "9dcb7f46d2628d7d2d91335aea6d5219"
server_key_iv: "1c472c1bad968088b3957b735ccb942b"
server_key_version: "encryptionkey@0"
I20260923 20:08:52.456035 2902 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260923 20:08:52.459645 2917 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:08:52.461019 2902 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260923 20:08:52.461229 2902 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790194121577027-1706-0/minicluster-data/ts-0/data,/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790194121577027-1706-0/minicluster-data/ts-0/wal
uuid: "f1669496db8e44798ae7354b2d1f315b"
format_stamp: "Formatted at 2026-09-23 20:08:52 on dist-test-slave-th4g"
server_key: "9dcb7f46d2628d7d2d91335aea6d5219"
server_key_iv: "1c472c1bad968088b3957b735ccb942b"
server_key_version: "encryptionkey@0"
I20260923 20:08:52.461565 2902 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790194121577027-1706-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790194121577027-1706-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790194121577027-1706-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:08:52.489552 2902 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:08:52.493721 2902 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:08:52.494033 2902 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:08:52.496663 2902 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 20:08:52.496790 2902 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 20:08:52.496956 2902 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 20:08:52.497026 2902 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 23 20:08:52 dist-test-slave-th4g krb5kdc[2786](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194132, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.170.129@KRBTEST.COM for kudu/127.1.170.190@KRBTEST.COM
I20260923 20:08:52.522043 2926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:08:52.497524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.170.129:39745 (local address 127.1.170.190:39075)
0923 20:08:52.498002 (+ 478us) server_negotiation.cc:206] Beginning negotiation
0923 20:08:52.498008 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:08:52.499325 (+ 1317us) server_negotiation.cc:407] Connection header received
0923 20:08:52.502467 (+ 3142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:08:52.502475 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:08:52.502572 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:08:52.502718 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:08:52.504875 (+ 2157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:08:52.506060 (+ 1185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:08:52.508488 (+ 2428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:08:52.508965 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:08:52.509114 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:08:52.514398 (+ 5284us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:08:52.514403 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:08:52.514407 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:08:52.514438 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:08:52.517720 (+ 3282us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:08:52.518909 (+ 1189us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:08:52.518917 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:08:52.518920 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:08:52.519073 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:08:52.519706 (+ 633us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:08:52.519714 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:08:52.519720 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:08:52.520183 (+ 463us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:08:52.520375 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0923 20:08:52.521131 (+ 756us) server_negotiation.cc:299] Negotiation successful
0923 20:08:52.521442 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":335,"thread_start_us":236,"threads_started":1}
I20260923 20:08:52.522686 2927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:08:52.498249 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.190:39075 (local address 127.1.170.129:39745)
0923 20:08:52.499136 (+ 887us) negotiation.cc:107] Waiting for socket to connect
0923 20:08:52.499189 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0923 20:08:52.502070 (+ 2881us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:08:52.502984 (+ 914us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:08:52.502992 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:08:52.503730 (+ 738us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:08:52.504676 (+ 946us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:08:52.504700 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:08:52.506231 (+ 1531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:08:52.506273 (+ 42us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:08:52.508115 (+ 1842us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:08:52.508153 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:08:52.509255 (+ 1102us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:08:52.509263 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:08:52.509625 (+ 362us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:08:52.510546 (+ 921us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:08:52.510588 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:08:52.514141 (+ 3553us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:08:52.517903 (+ 3762us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:08:52.517916 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:08:52.517936 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:08:52.518735 (+ 799us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:08:52.519214 (+ 479us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:08:52.519219 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:08:52.519223 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:08:52.519538 (+ 315us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:08:52.520335 (+ 797us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:08:52.520341 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:08:52.520713 (+ 372us) client_negotiation.cc:769] Sending connection context
0923 20:08:52.520943 (+ 230us) client_negotiation.cc:240] Negotiation successful
0923 20:08:52.521349 (+ 406us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":581,"thread_start_us":351,"threads_started":1}
I20260923 20:08:52.544535 2926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:08:52.530076 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.170.129:43905 (local address 127.1.170.190:39075)
0923 20:08:52.530380 (+ 304us) server_negotiation.cc:206] Beginning negotiation
0923 20:08:52.530389 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 20:08:52.530660 (+ 271us) server_negotiation.cc:407] Connection header received
0923 20:08:52.530905 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:08:52.530914 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:08:52.531018 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:08:52.531147 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:08:52.532080 (+ 933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:08:52.533523 (+ 1443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:08:52.535472 (+ 1949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:08:52.536124 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:08:52.536334 (+ 210us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:08:52.538419 (+ 2085us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:08:52.538428 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:08:52.538434 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:08:52.538474 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:08:52.541656 (+ 3182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:08:52.542424 (+ 768us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:08:52.542431 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:08:52.542439 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:08:52.542549 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:08:52.542931 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:08:52.542936 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:08:52.542941 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:08:52.543334 (+ 393us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:08:52.543484 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0923 20:08:52.543881 (+ 397us) server_negotiation.cc:299] Negotiation successful
0923 20:08:52.544092 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118}
I20260923 20:08:52.544606 2996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:08:52.529953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.190:39075 (local address 127.1.170.129:43905)
0923 20:08:52.530530 (+ 577us) negotiation.cc:107] Waiting for socket to connect
0923 20:08:52.530545 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0923 20:08:52.530762 (+ 217us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:08:52.531166 (+ 404us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:08:52.531169 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:08:52.531638 (+ 469us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:08:52.531936 (+ 298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:08:52.531944 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:08:52.533681 (+ 1737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:08:52.533690 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:08:52.535312 (+ 1622us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:08:52.535334 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:08:52.536429 (+ 1095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:08:52.536436 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:08:52.536576 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:08:52.537220 (+ 644us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:08:52.537237 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:08:52.538150 (+ 913us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:08:52.541796 (+ 3646us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:08:52.541803 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:08:52.541807 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:08:52.542291 (+ 484us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:08:52.542659 (+ 368us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:08:52.542668 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:08:52.542672 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:08:52.542800 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:08:52.543455 (+ 655us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:08:52.543460 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:08:52.543641 (+ 181us) client_negotiation.cc:769] Sending connection context
0923 20:08:52.543772 (+ 131us) client_negotiation.cc:240] Negotiation successful
0923 20:08:52.544092 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":449,"thread_start_us":430,"threads_started":1}
I20260923 20:08:52.556221 2902 rpc_server.cc:307] RPC server started. Bound to: 127.1.170.129:38261
I20260923 20:08:52.556269 3037 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.170.129:38261 every 8 connection(s)
I20260923 20:08:52.557425 2902 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790194121577027-1706-0/minicluster-data/ts-0/data/info.pb
I20260923 20:08:52.558115 1706 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqUuku6/build/asan/bin/kudu as pid 2902
I20260923 20:08:52.558358 1706 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790194121577027-1706-0/minicluster-data/ts-0/wal/instance
I20260923 20:08:52.559140 1706 external_mini_cluster.cc:1428] Setting key b7e1556cf848a75707bb1970c0477833
I20260923 20:08:52.559875 3038 heartbeater.cc:347] Connected to a master server at 127.1.170.190:39075
I20260923 20:08:52.560122 3038 heartbeater.cc:464] Registering TS with master...
I20260923 20:08:52.560799 3038 heartbeater.cc:511] Master 127.1.170.190:39075 requested a full tablet report, sending...
I20260923 20:08:52.563128 2831 ts_manager.cc:194] Registered new tserver with Master: f1669496db8e44798ae7354b2d1f315b (127.1.170.129:38261)
I20260923 20:08:52.565358 2831 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.170.129@KRBTEST.COM'} at 127.1.170.129:39745
WARNING: no policy specified for kudu/127.1.170.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.170.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.170.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790194121577027-1706-0/krb5kdc/kudu_127.1.170.130.keytab.
Entry for principal kudu/127.1.170.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790194121577027-1706-0/krb5kdc/kudu_127.1.170.130.keytab.
I20260923 20:08:52.601123 1706 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqUuku6/build/asan/bin/kudu
/tmp/dist-test-taskqUuku6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790194121577027-1706-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790194121577027-1706-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790194121577027-1706-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790194121577027-1706-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.170.130:0
--local_ip_for_outbound_sockets=127.1.170.130
--webserver_interface=127.1.170.130
--webserver_port=0
--tserver_master_addrs=127.1.170.190:39075
--keytab_file=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790194121577027-1706-0/krb5kdc/kudu_127.1.170.130.keytab
--principal=kudu/127.1.170.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.170.148:41579
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790194121577027-1706-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790194121577027-1706-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790194121577027-1706-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:08:52.928191 3042 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:08:52.928476 3042 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:08:52.928632 3042 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:08:52.936424 3042 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:08:52.936542 3042 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:08:52.936687 3042 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260923 20:08:52.936779 3042 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.170.130
I20260923 20:08:52.947094 3042 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.170.148:41579
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790194121577027-1706-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790194121577027-1706-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790194121577027-1706-0/krb5kdc/kudu_127.1.170.130.keytab
--principal=kudu/127.1.170.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.170.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790194121577027-1706-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.170.130
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.1.170.190:39075
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.170.130
--log_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790194121577027-1706-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:08:52.948505 3042 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:08:52.949831 3042 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:08:52.963455 3048 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:08:52.963455 3050 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:08:52.963496 3047 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:08:52.970209 3042 server_base.cc:1034] running on GCE node
I20260923 20:08:52.970903 3042 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:08:52.971902 3042 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:08:52.973174 3042 hybrid_clock.cc:648] HybridClock initialized: now 1790194132973094 us; error 49 us; skew 500 ppm
I20260923 20:08:52.973618 3042 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:08:52 dist-test-slave-th4g krb5kdc[2786](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194132, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.170.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:08:52.977156 3042 init.cc:377] Logged in from keytab as kudu/127.1.170.130@KRBTEST.COM (short username kudu)
I20260923 20:08:52.978999 3042 webserver.cc:460] Webserver started at http://127.1.170.130:35433/ using document root <none> and password file <none>
I20260923 20:08:52.979804 3042 fs_manager.cc:359] Metadata directory not provided
I20260923 20:08:52.979949 3042 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:08:52.980304 3042 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:08:52.983534 3042 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790194121577027-1706-0/minicluster-data/ts-1/data/instance:
uuid: "b56de26098334552bb9ae268d771c311"
format_stamp: "Formatted at 2026-09-23 20:08:52 on dist-test-slave-th4g"
server_key: "c01d984ae5afc97cd813ad02e23d891c"
server_key_iv: "75112e3b42e48a9165fc49b30c719f4e"
server_key_version: "encryptionkey@0"
I20260923 20:08:52.984432 3042 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790194121577027-1706-0/minicluster-data/ts-1/wal/instance:
uuid: "b56de26098334552bb9ae268d771c311"
format_stamp: "Formatted at 2026-09-23 20:08:52 on dist-test-slave-th4g"
server_key: "c01d984ae5afc97cd813ad02e23d891c"
server_key_iv: "75112e3b42e48a9165fc49b30c719f4e"
server_key_version: "encryptionkey@0"
I20260923 20:08:52.990368 3042 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.004s
I20260923 20:08:52.994596 3057 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:08:52.996083 3042 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260923 20:08:52.996292 3042 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790194121577027-1706-0/minicluster-data/ts-1/data,/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790194121577027-1706-0/minicluster-data/ts-1/wal
uuid: "b56de26098334552bb9ae268d771c311"
format_stamp: "Formatted at 2026-09-23 20:08:52 on dist-test-slave-th4g"
server_key: "c01d984ae5afc97cd813ad02e23d891c"
server_key_iv: "75112e3b42e48a9165fc49b30c719f4e"
server_key_version: "encryptionkey@0"
I20260923 20:08:52.996544 3042 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790194121577027-1706-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790194121577027-1706-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790194121577027-1706-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:08:53.019133 3042 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:08:53.023540 3042 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:08:53.023815 3042 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:08:53.026850 3042 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 20:08:53.026961 3042 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 20:08:53.027108 3042 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 20:08:53.027212 3042 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
Sep 23 20:08:53 dist-test-slave-th4g krb5kdc[2786](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194132, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.170.130@KRBTEST.COM for kudu/127.1.170.190@KRBTEST.COM
I20260923 20:08:53.050760 2926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:08:53.027324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.170.130:55739 (local address 127.1.170.190:39075)
0923 20:08:53.027528 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0923 20:08:53.027537 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 20:08:53.029020 (+ 1483us) server_negotiation.cc:407] Connection header received
0923 20:08:53.031351 (+ 2331us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:08:53.031357 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:08:53.031472 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:08:53.031582 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:08:53.033815 (+ 2233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:08:53.034890 (+ 1075us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:08:53.036961 (+ 2071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:08:53.037404 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:08:53.037545 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:08:53.042648 (+ 5103us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:08:53.042653 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:08:53.042656 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:08:53.042688 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:08:53.045926 (+ 3238us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:08:53.047123 (+ 1197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:08:53.047130 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:08:53.047134 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:08:53.047297 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:08:53.047964 (+ 667us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:08:53.047973 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:08:53.047978 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:08:53.048423 (+ 445us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:08:53.048573 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0923 20:08:53.049980 (+ 1407us) server_negotiation.cc:299] Negotiation successful
0923 20:08:53.050259 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260923 20:08:53.054064 3066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:08:53.028008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.190:39075 (local address 127.1.170.130:55739)
0923 20:08:53.028851 (+ 843us) negotiation.cc:107] Waiting for socket to connect
0923 20:08:53.028893 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0923 20:08:53.031039 (+ 2146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:08:53.031820 (+ 781us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:08:53.031831 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:08:53.032631 (+ 800us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:08:53.033542 (+ 911us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:08:53.033573 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:08:53.035069 (+ 1496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:08:53.035097 (+ 28us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:08:53.036818 (+ 1721us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:08:53.036830 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:08:53.037652 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:08:53.037660 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:08:53.037956 (+ 296us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:08:53.038886 (+ 930us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:08:53.038929 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:08:53.042296 (+ 3367us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:08:53.046098 (+ 3802us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:08:53.046107 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:08:53.046125 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:08:53.046924 (+ 799us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:08:53.047484 (+ 560us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:08:53.047491 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:08:53.047497 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:08:53.047803 (+ 306us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:08:53.048742 (+ 939us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:08:53.048762 (+ 20us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:08:53.049397 (+ 635us) client_negotiation.cc:769] Sending connection context
0923 20:08:53.051228 (+ 1831us) client_negotiation.cc:240] Negotiation successful
0923 20:08:53.051983 (+ 755us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":614,"thread_start_us":414,"threads_started":1}
I20260923 20:08:53.072628 3042 rpc_server.cc:307] RPC server started. Bound to: 127.1.170.130:33471
I20260923 20:08:53.072580 3176 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.170.130:33471 every 8 connection(s)
I20260923 20:08:53.074390 3042 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790194121577027-1706-0/minicluster-data/ts-1/data/info.pb
I20260923 20:08:53.076855 3177 heartbeater.cc:347] Connected to a master server at 127.1.170.190:39075
I20260923 20:08:53.077134 3177 heartbeater.cc:464] Registering TS with master...
I20260923 20:08:53.077689 2926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:08:53.062161 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.170.130:47171 (local address 127.1.170.190:39075)
0923 20:08:53.062413 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0923 20:08:53.062423 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0923 20:08:53.063072 (+ 649us) server_negotiation.cc:407] Connection header received
0923 20:08:53.063402 (+ 330us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:08:53.063410 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:08:53.063501 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:08:53.063636 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:08:53.064545 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:08:53.065976 (+ 1431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:08:53.067328 (+ 1352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:08:53.068082 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:08:53.068237 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:08:53.070736 (+ 2499us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:08:53.070742 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:08:53.070747 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:08:53.070783 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:08:53.074404 (+ 3621us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:08:53.075382 (+ 978us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:08:53.075388 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:08:53.075391 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:08:53.075587 (+ 196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:08:53.076020 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:08:53.076028 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:08:53.076034 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:08:53.076448 (+ 414us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:08:53.076605 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0923 20:08:53.077119 (+ 514us) server_negotiation.cc:299] Negotiation successful
0923 20:08:53.077344 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260923 20:08:53.077947 3156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:08:53.062284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.190:39075 (local address 127.1.170.130:47171)
0923 20:08:53.062934 (+ 650us) negotiation.cc:107] Waiting for socket to connect
0923 20:08:53.062958 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0923 20:08:53.063136 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:08:53.063698 (+ 562us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:08:53.063701 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:08:53.064103 (+ 402us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:08:53.064392 (+ 289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:08:53.064402 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:08:53.066119 (+ 1717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:08:53.066124 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:08:53.067182 (+ 1058us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:08:53.067195 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:08:53.068402 (+ 1207us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:08:53.068416 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:08:53.068625 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:08:53.069496 (+ 871us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:08:53.069513 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:08:53.070540 (+ 1027us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:08:53.074588 (+ 4048us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:08:53.074593 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:08:53.074596 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:08:53.075206 (+ 610us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:08:53.075731 (+ 525us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:08:53.075735 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:08:53.075738 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:08:53.075906 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:08:53.076603 (+ 697us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:08:53.076609 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:08:53.076819 (+ 210us) client_negotiation.cc:769] Sending connection context
0923 20:08:53.077003 (+ 184us) client_negotiation.cc:240] Negotiation successful
0923 20:08:53.077419 (+ 416us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":500,"thread_start_us":351,"threads_started":1}
I20260923 20:08:53.078029 3177 heartbeater.cc:511] Master 127.1.170.190:39075 requested a full tablet report, sending...
I20260923 20:08:53.079197 1706 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqUuku6/build/asan/bin/kudu as pid 3042
I20260923 20:08:53.079413 1706 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790194121577027-1706-0/minicluster-data/ts-1/wal/instance
I20260923 20:08:53.079632 2830 ts_manager.cc:194] Registered new tserver with Master: b56de26098334552bb9ae268d771c311 (127.1.170.130:33471)
I20260923 20:08:53.079949 1706 external_mini_cluster.cc:1428] Setting key ea37b260cf85e356f2398728c817a336
I20260923 20:08:53.080732 2830 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.170.130@KRBTEST.COM'} at 127.1.170.130:55739
WARNING: no policy specified for kudu/127.1.170.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.170.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.170.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790194121577027-1706-0/krb5kdc/kudu_127.1.170.131.keytab.
Entry for principal kudu/127.1.170.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790194121577027-1706-0/krb5kdc/kudu_127.1.170.131.keytab.
I20260923 20:08:53.127070 1706 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqUuku6/build/asan/bin/kudu
/tmp/dist-test-taskqUuku6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790194121577027-1706-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790194121577027-1706-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790194121577027-1706-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790194121577027-1706-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.170.131:0
--local_ip_for_outbound_sockets=127.1.170.131
--webserver_interface=127.1.170.131
--webserver_port=0
--tserver_master_addrs=127.1.170.190:39075
--keytab_file=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790194121577027-1706-0/krb5kdc/kudu_127.1.170.131.keytab
--principal=kudu/127.1.170.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.170.148:41579
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790194121577027-1706-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790194121577027-1706-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790194121577027-1706-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:08:53.472635 3181 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:08:53.472965 3181 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:08:53.473073 3181 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:08:53.481556 3181 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:08:53.481786 3181 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:08:53.481992 3181 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260923 20:08:53.482167 3181 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.170.131
I20260923 20:08:53.493589 3181 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.170.148:41579
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790194121577027-1706-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790194121577027-1706-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790194121577027-1706-0/krb5kdc/kudu_127.1.170.131.keytab
--principal=kudu/127.1.170.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.170.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790194121577027-1706-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.170.131
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.1.170.190:39075
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.170.131
--log_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790194121577027-1706-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:08:53.495056 3181 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:08:53.496492 3181 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:08:53.512182 3189 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:08:53.512115 3187 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:08:53.512086 3186 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:08:53.512909 3181 server_base.cc:1034] running on GCE node
I20260923 20:08:53.513768 3181 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:08:53.515264 3181 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:08:53.516901 3181 hybrid_clock.cc:648] HybridClock initialized: now 1790194133516757 us; error 106 us; skew 500 ppm
I20260923 20:08:53.517310 3181 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:08:53 dist-test-slave-th4g krb5kdc[2786](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194133, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.170.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:08:53.521852 3181 init.cc:377] Logged in from keytab as kudu/127.1.170.131@KRBTEST.COM (short username kudu)
I20260923 20:08:53.523770 3181 webserver.cc:460] Webserver started at http://127.1.170.131:39555/ using document root <none> and password file <none>
I20260923 20:08:53.524610 3181 fs_manager.cc:359] Metadata directory not provided
I20260923 20:08:53.524772 3181 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:08:53.525183 3181 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:08:53.528395 3181 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790194121577027-1706-0/minicluster-data/ts-2/data/instance:
uuid: "42013bee84ed41b59532dc3b7fd90adb"
format_stamp: "Formatted at 2026-09-23 20:08:53 on dist-test-slave-th4g"
server_key: "13776d006f2f6f4e5cef618faf446c2a"
server_key_iv: "39b835194632d8a2435254ae4f4639e4"
server_key_version: "encryptionkey@0"
I20260923 20:08:53.529284 3181 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790194121577027-1706-0/minicluster-data/ts-2/wal/instance:
uuid: "42013bee84ed41b59532dc3b7fd90adb"
format_stamp: "Formatted at 2026-09-23 20:08:53 on dist-test-slave-th4g"
server_key: "13776d006f2f6f4e5cef618faf446c2a"
server_key_iv: "39b835194632d8a2435254ae4f4639e4"
server_key_version: "encryptionkey@0"
I20260923 20:08:53.537467 3181 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.009s sys 0.001s
I20260923 20:08:53.542374 3196 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:08:53.544811 3181 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.003s sys 0.000s
I20260923 20:08:53.545128 3181 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790194121577027-1706-0/minicluster-data/ts-2/data,/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790194121577027-1706-0/minicluster-data/ts-2/wal
uuid: "42013bee84ed41b59532dc3b7fd90adb"
format_stamp: "Formatted at 2026-09-23 20:08:53 on dist-test-slave-th4g"
server_key: "13776d006f2f6f4e5cef618faf446c2a"
server_key_iv: "39b835194632d8a2435254ae4f4639e4"
server_key_version: "encryptionkey@0"
I20260923 20:08:53.545428 3181 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790194121577027-1706-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790194121577027-1706-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790194121577027-1706-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:08:53.566023 3181 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:08:53.568210 3038 heartbeater.cc:503] Master 127.1.170.190:39075 was elected leader, sending a full tablet report...
I20260923 20:08:53.572057 3181 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:08:53.572412 3181 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:08:53.575877 3181 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 20:08:53.575995 3181 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 20:08:53.576159 3181 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 20:08:53.576236 3181 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 23 20:08:53 dist-test-slave-th4g krb5kdc[2786](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194133, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.170.131@KRBTEST.COM for kudu/127.1.170.190@KRBTEST.COM
I20260923 20:08:53.604645 2926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:08:53.577221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.170.131:48219 (local address 127.1.170.190:39075)
0923 20:08:53.577432 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0923 20:08:53.577439 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:08:53.579003 (+ 1564us) server_negotiation.cc:407] Connection header received
0923 20:08:53.581326 (+ 2323us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:08:53.581333 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:08:53.581418 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:08:53.581479 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:08:53.584734 (+ 3255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:08:53.585616 (+ 882us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:08:53.588938 (+ 3322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:08:53.589595 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:08:53.589768 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:08:53.596626 (+ 6858us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:08:53.596640 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:08:53.596647 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:08:53.596719 (+ 72us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:08:53.600334 (+ 3615us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:08:53.601477 (+ 1143us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:08:53.601485 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:08:53.601490 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:08:53.601697 (+ 207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:08:53.602337 (+ 640us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:08:53.602342 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:08:53.602347 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:08:53.602790 (+ 443us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:08:53.602985 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0923 20:08:53.604034 (+ 1049us) server_negotiation.cc:299] Negotiation successful
0923 20:08:53.604271 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260923 20:08:53.605696 3205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:08:53.577735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.190:39075 (local address 127.1.170.131:48219)
0923 20:08:53.578788 (+ 1053us) negotiation.cc:107] Waiting for socket to connect
0923 20:08:53.578839 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0923 20:08:53.580904 (+ 2065us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:08:53.582644 (+ 1740us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:08:53.582662 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:08:53.583537 (+ 875us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:08:53.584358 (+ 821us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:08:53.584383 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:08:53.586000 (+ 1617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:08:53.586038 (+ 38us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:08:53.588656 (+ 2618us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:08:53.588682 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:08:53.589993 (+ 1311us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:08:53.590006 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:08:53.590435 (+ 429us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:08:53.591669 (+ 1234us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:08:53.591706 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:08:53.596153 (+ 4447us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:08:53.600537 (+ 4384us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:08:53.600545 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:08:53.600562 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:08:53.601329 (+ 767us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:08:53.601829 (+ 500us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:08:53.601840 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:08:53.601843 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:08:53.602184 (+ 341us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:08:53.602983 (+ 799us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:08:53.602996 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:08:53.603529 (+ 533us) client_negotiation.cc:769] Sending connection context
0923 20:08:53.603913 (+ 384us) client_negotiation.cc:240] Negotiation successful
0923 20:08:53.604499 (+ 586us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":748,"thread_start_us":457,"threads_started":1}
I20260923 20:08:53.632423 3272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:08:53.612410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.190:39075 (local address 127.1.170.131:55631)
0923 20:08:53.617584 (+ 5174us) negotiation.cc:107] Waiting for socket to connect
0923 20:08:53.617609 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0923 20:08:53.617823 (+ 214us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:08:53.618500 (+ 677us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:08:53.618507 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:08:53.619291 (+ 784us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:08:53.620246 (+ 955us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:08:53.620268 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:08:53.621621 (+ 1353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:08:53.621629 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:08:53.622856 (+ 1227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:08:53.622868 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:08:53.623852 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:08:53.623858 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:08:53.624041 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:08:53.624925 (+ 884us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:08:53.624956 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:08:53.626291 (+ 1335us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:08:53.629561 (+ 3270us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:08:53.629567 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:08:53.629571 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:08:53.630074 (+ 503us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:08:53.630526 (+ 452us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:08:53.630536 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:08:53.630540 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:08:53.630687 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:08:53.631451 (+ 764us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:08:53.631458 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:08:53.631654 (+ 196us) client_negotiation.cc:769] Sending connection context
0923 20:08:53.631793 (+ 139us) client_negotiation.cc:240] Negotiation successful
0923 20:08:53.632060 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4950,"thread_start_us":4737,"threads_started":1}
I20260923 20:08:53.633260 2926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:08:53.612482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.170.131:55631 (local address 127.1.170.190:39075)
0923 20:08:53.612730 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0923 20:08:53.612739 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 20:08:53.617749 (+ 5010us) server_negotiation.cc:407] Connection header received
0923 20:08:53.618041 (+ 292us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:08:53.618048 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:08:53.618158 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:08:53.618239 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:08:53.620459 (+ 2220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:08:53.621387 (+ 928us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:08:53.623019 (+ 1632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:08:53.623618 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:08:53.623774 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:08:53.626543 (+ 2769us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:08:53.626548 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:08:53.626551 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:08:53.626581 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:08:53.629406 (+ 2825us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:08:53.630205 (+ 799us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:08:53.630211 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:08:53.630214 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:08:53.630384 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:08:53.630811 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:08:53.630814 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:08:53.630816 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:08:53.631286 (+ 470us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:08:53.632693 (+ 1407us) server_negotiation.cc:1035] Waiting for connection context
0923 20:08:53.632804 (+ 111us) server_negotiation.cc:299] Negotiation successful
0923 20:08:53.632968 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260923 20:08:53.638826 3181 rpc_server.cc:307] RPC server started. Bound to: 127.1.170.131:44771
I20260923 20:08:53.638864 3315 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.170.131:44771 every 8 connection(s)
I20260923 20:08:53.640084 3181 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790194121577027-1706-0/minicluster-data/ts-2/data/info.pb
I20260923 20:08:53.642409 3316 heartbeater.cc:347] Connected to a master server at 127.1.170.190:39075
I20260923 20:08:53.642652 3316 heartbeater.cc:464] Registering TS with master...
I20260923 20:08:53.643337 3316 heartbeater.cc:511] Master 127.1.170.190:39075 requested a full tablet report, sending...
I20260923 20:08:53.646872 2830 ts_manager.cc:194] Registered new tserver with Master: 42013bee84ed41b59532dc3b7fd90adb (127.1.170.131:44771)
I20260923 20:08:53.648438 1706 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqUuku6/build/asan/bin/kudu as pid 3181
I20260923 20:08:53.648872 1706 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790194121577027-1706-0/minicluster-data/ts-2/wal/instance
I20260923 20:08:53.649070 2830 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.170.131@KRBTEST.COM'} at 127.1.170.131:48219
I20260923 20:08:53.649610 1706 external_mini_cluster.cc:1428] Setting key 395d472a4505456476c54ba5856e4600
I20260923 20:08:53.654398 1706 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 23 20:08:53 dist-test-slave-th4g krb5kdc[2786](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194133, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 23 20:08:53 dist-test-slave-th4g krb5kdc[2786](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194133, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.1.170.190@KRBTEST.COM
I20260923 20:08:53.703495 2926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:08:53.688737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42432 (local address 127.1.170.190:39075)
0923 20:08:53.688959 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0923 20:08:53.688965 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:08:53.689184 (+ 219us) server_negotiation.cc:407] Connection header received
0923 20:08:53.689386 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:08:53.689390 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:08:53.689460 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:08:53.689542 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:08:53.691070 (+ 1528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:08:53.691836 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:08:53.693535 (+ 1699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:08:53.694031 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:08:53.694127 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:08:53.698064 (+ 3937us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:08:53.698069 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:08:53.698072 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:08:53.698105 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:08:53.700767 (+ 2662us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:08:53.701496 (+ 729us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:08:53.701505 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:08:53.701512 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:08:53.701663 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:08:53.702073 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:08:53.702077 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:08:53.702079 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:08:53.702373 (+ 294us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:08:53.702564 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0923 20:08:53.702865 (+ 301us) server_negotiation.cc:299] Negotiation successful
0923 20:08:53.703037 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260923 20:08:53.707789 2830 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42432:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 20:08:53.714903 2830 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 20:08:53.743111 3329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:08:53.729320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.130:33471 (local address 127.0.0.1:51168)
0923 20:08:53.730219 (+ 899us) negotiation.cc:107] Waiting for socket to connect
0923 20:08:53.730265 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0923 20:08:53.730438 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:08:53.735257 (+ 4819us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:08:53.735263 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:08:53.735294 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:08:53.735795 (+ 501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:08:53.735806 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:08:53.738250 (+ 2444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:08:53.738256 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:08:53.740664 (+ 2408us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:08:53.740673 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:08:53.741940 (+ 1267us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:08:53.741948 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:08:53.742154 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:08:53.742217 (+ 63us) client_negotiation.cc:769] Sending connection context
0923 20:08:53.742358 (+ 141us) client_negotiation.cc:240] Negotiation successful
0923 20:08:53.742573 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":514,"mutex_wait_us":139,"spinlock_wait_cycles":19584,"thread_start_us":598,"threads_started":1}
I20260923 20:08:53.743551 3330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:08:53.729422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.129:38261 (local address 127.0.0.1:52274)
0923 20:08:53.730219 (+ 797us) negotiation.cc:107] Waiting for socket to connect
0923 20:08:53.730264 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0923 20:08:53.730438 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:08:53.735111 (+ 4673us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:08:53.735119 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:08:53.735156 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:08:53.735688 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:08:53.735702 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:08:53.738299 (+ 2597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:08:53.738303 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:08:53.740722 (+ 2419us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:08:53.740739 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:08:53.742586 (+ 1847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:08:53.742597 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:08:53.742815 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:08:53.742842 (+ 27us) client_negotiation.cc:769] Sending connection context
0923 20:08:53.742972 (+ 130us) client_negotiation.cc:240] Negotiation successful
0923 20:08:53.743141 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":580,"mutex_wait_us":120,"spinlock_wait_cycles":3328,"thread_start_us":623,"threads_started":1}
I20260923 20:08:53.743795 3331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:08:53.729741 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51168 (local address 127.1.170.130:33471)
0923 20:08:53.734660 (+ 4919us) server_negotiation.cc:206] Beginning negotiation
0923 20:08:53.734668 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 20:08:53.734691 (+ 23us) server_negotiation.cc:407] Connection header received
0923 20:08:53.734772 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:08:53.734778 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:08:53.735055 (+ 277us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:08:53.736019 (+ 964us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:08:53.736542 (+ 523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:08:53.738110 (+ 1568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:08:53.740815 (+ 2705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:08:53.741706 (+ 891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:08:53.741867 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:08:53.742114 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0923 20:08:53.743103 (+ 989us) server_negotiation.cc:299] Negotiation successful
0923 20:08:53.743417 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4776,"thread_start_us":4622,"threads_started":1}
I20260923 20:08:53.743949 3332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:08:53.729862 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52274 (local address 127.1.170.129:38261)
0923 20:08:53.734369 (+ 4507us) server_negotiation.cc:206] Beginning negotiation
0923 20:08:53.734381 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0923 20:08:53.734418 (+ 37us) server_negotiation.cc:407] Connection header received
0923 20:08:53.734529 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:08:53.734538 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:08:53.734836 (+ 298us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:08:53.735946 (+ 1110us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:08:53.736559 (+ 613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:08:53.738119 (+ 1560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:08:53.740932 (+ 2813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:08:53.742364 (+ 1432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:08:53.742530 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:08:53.742752 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0923 20:08:53.743018 (+ 266us) server_negotiation.cc:299] Negotiation successful
0923 20:08:53.743220 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4305,"thread_start_us":4175,"threads_started":1}
I20260923 20:08:53.744190 3334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:08:53.731093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35654 (local address 127.1.170.131:44771)
0923 20:08:53.732147 (+ 1054us) server_negotiation.cc:206] Beginning negotiation
0923 20:08:53.732159 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0923 20:08:53.734702 (+ 2543us) server_negotiation.cc:407] Connection header received
0923 20:08:53.735011 (+ 309us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:08:53.735018 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:08:53.735230 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:08:53.735415 (+ 185us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:08:53.736380 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:08:53.737468 (+ 1088us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:08:53.740425 (+ 2957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:08:53.741958 (+ 1533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:08:53.742085 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:08:53.742303 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0923 20:08:53.743694 (+ 1391us) server_negotiation.cc:299] Negotiation successful
0923 20:08:53.743870 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":840,"thread_start_us":575,"threads_started":1}
I20260923 20:08:53.744416 3333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:08:53.730752 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.131:44771 (local address 127.0.0.1:35654)
0923 20:08:53.734354 (+ 3602us) negotiation.cc:107] Waiting for socket to connect
0923 20:08:53.734385 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0923 20:08:53.734821 (+ 436us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:08:53.735819 (+ 998us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:08:53.735822 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:08:53.735845 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:08:53.736189 (+ 344us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:08:53.736197 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:08:53.737627 (+ 1430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:08:53.737632 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:08:53.740235 (+ 2603us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:08:53.740251 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:08:53.742264 (+ 2013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:08:53.742270 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:08:53.742443 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:08:53.742472 (+ 29us) client_negotiation.cc:769] Sending connection context
0923 20:08:53.742551 (+ 79us) client_negotiation.cc:240] Negotiation successful
0923 20:08:53.742680 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3467,"mutex_wait_us":58,"thread_start_us":3348,"threads_started":1}
I20260923 20:08:53.747454 2968 tablet_service.cc:1467] Processing CreateTablet for tablet 47ebc61a3bfb40e8b7b6d350c957347d (DEFAULT_TABLE table=test-table [id=d14c6f0609514b4da971b482e687b415]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 20:08:53.747650 3107 tablet_service.cc:1467] Processing CreateTablet for tablet 47ebc61a3bfb40e8b7b6d350c957347d (DEFAULT_TABLE table=test-table [id=d14c6f0609514b4da971b482e687b415]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 20:08:53.747952 3246 tablet_service.cc:1467] Processing CreateTablet for tablet 47ebc61a3bfb40e8b7b6d350c957347d (DEFAULT_TABLE table=test-table [id=d14c6f0609514b4da971b482e687b415]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 20:08:53.748993 2968 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 47ebc61a3bfb40e8b7b6d350c957347d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:08:53.749011 3107 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 47ebc61a3bfb40e8b7b6d350c957347d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:08:53.749195 3246 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 47ebc61a3bfb40e8b7b6d350c957347d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:08:53.766585 3336 tablet_bootstrap.cc:492] T 47ebc61a3bfb40e8b7b6d350c957347d P 42013bee84ed41b59532dc3b7fd90adb: Bootstrap starting.
I20260923 20:08:53.767633 3335 tablet_bootstrap.cc:492] T 47ebc61a3bfb40e8b7b6d350c957347d P f1669496db8e44798ae7354b2d1f315b: Bootstrap starting.
I20260923 20:08:53.768878 3337 tablet_bootstrap.cc:492] T 47ebc61a3bfb40e8b7b6d350c957347d P b56de26098334552bb9ae268d771c311: Bootstrap starting.
I20260923 20:08:53.770954 3336 tablet_bootstrap.cc:654] T 47ebc61a3bfb40e8b7b6d350c957347d P 42013bee84ed41b59532dc3b7fd90adb: Neither blocks nor log segments found. Creating new log.
I20260923 20:08:53.771051 3335 tablet_bootstrap.cc:654] T 47ebc61a3bfb40e8b7b6d350c957347d P f1669496db8e44798ae7354b2d1f315b: Neither blocks nor log segments found. Creating new log.
I20260923 20:08:53.771996 3337 tablet_bootstrap.cc:654] T 47ebc61a3bfb40e8b7b6d350c957347d P b56de26098334552bb9ae268d771c311: Neither blocks nor log segments found. Creating new log.
I20260923 20:08:53.772317 3336 log.cc:824] T 47ebc61a3bfb40e8b7b6d350c957347d P 42013bee84ed41b59532dc3b7fd90adb: Log is configured to *not* fsync() on all Append() calls
I20260923 20:08:53.772317 3335 log.cc:824] T 47ebc61a3bfb40e8b7b6d350c957347d P f1669496db8e44798ae7354b2d1f315b: Log is configured to *not* fsync() on all Append() calls
I20260923 20:08:53.773247 3337 log.cc:824] T 47ebc61a3bfb40e8b7b6d350c957347d P b56de26098334552bb9ae268d771c311: Log is configured to *not* fsync() on all Append() calls
I20260923 20:08:53.775769 3335 tablet_bootstrap.cc:492] T 47ebc61a3bfb40e8b7b6d350c957347d P f1669496db8e44798ae7354b2d1f315b: No bootstrap required, opened a new log
I20260923 20:08:53.775768 3336 tablet_bootstrap.cc:492] T 47ebc61a3bfb40e8b7b6d350c957347d P 42013bee84ed41b59532dc3b7fd90adb: No bootstrap required, opened a new log
I20260923 20:08:53.776063 3336 ts_tablet_manager.cc:1397] T 47ebc61a3bfb40e8b7b6d350c957347d P 42013bee84ed41b59532dc3b7fd90adb: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260923 20:08:53.776063 3335 ts_tablet_manager.cc:1397] T 47ebc61a3bfb40e8b7b6d350c957347d P f1669496db8e44798ae7354b2d1f315b: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.000s
I20260923 20:08:53.776633 3337 tablet_bootstrap.cc:492] T 47ebc61a3bfb40e8b7b6d350c957347d P b56de26098334552bb9ae268d771c311: No bootstrap required, opened a new log
I20260923 20:08:53.776913 3337 ts_tablet_manager.cc:1397] T 47ebc61a3bfb40e8b7b6d350c957347d P b56de26098334552bb9ae268d771c311: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260923 20:08:53.786041 3337 raft_consensus.cc:348] T 47ebc61a3bfb40e8b7b6d350c957347d P b56de26098334552bb9ae268d771c311 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b56de26098334552bb9ae268d771c311" member_type: VOTER last_known_addr { host: "127.1.170.130" port: 33471 } } peers { permanent_uuid: "f1669496db8e44798ae7354b2d1f315b" member_type: VOTER last_known_addr { host: "127.1.170.129" port: 38261 } } peers { permanent_uuid: "42013bee84ed41b59532dc3b7fd90adb" member_type: VOTER last_known_addr { host: "127.1.170.131" port: 44771 } }
I20260923 20:08:53.786528 3337 raft_consensus.cc:374] T 47ebc61a3bfb40e8b7b6d350c957347d P b56de26098334552bb9ae268d771c311 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:08:53.786628 3337 raft_consensus.cc:729] T 47ebc61a3bfb40e8b7b6d350c957347d P b56de26098334552bb9ae268d771c311 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b56de26098334552bb9ae268d771c311, State: Initialized, Role: FOLLOWER
I20260923 20:08:53.787284 3337 consensus_queue.cc:260] T 47ebc61a3bfb40e8b7b6d350c957347d P b56de26098334552bb9ae268d771c311 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b56de26098334552bb9ae268d771c311" member_type: VOTER last_known_addr { host: "127.1.170.130" port: 33471 } } peers { permanent_uuid: "f1669496db8e44798ae7354b2d1f315b" member_type: VOTER last_known_addr { host: "127.1.170.129" port: 38261 } } peers { permanent_uuid: "42013bee84ed41b59532dc3b7fd90adb" member_type: VOTER last_known_addr { host: "127.1.170.131" port: 44771 } }
I20260923 20:08:53.788378 3177 heartbeater.cc:503] Master 127.1.170.190:39075 was elected leader, sending a full tablet report...
I20260923 20:08:53.788576 3335 raft_consensus.cc:348] T 47ebc61a3bfb40e8b7b6d350c957347d P f1669496db8e44798ae7354b2d1f315b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b56de26098334552bb9ae268d771c311" member_type: VOTER last_known_addr { host: "127.1.170.130" port: 33471 } } peers { permanent_uuid: "f1669496db8e44798ae7354b2d1f315b" member_type: VOTER last_known_addr { host: "127.1.170.129" port: 38261 } } peers { permanent_uuid: "42013bee84ed41b59532dc3b7fd90adb" member_type: VOTER last_known_addr { host: "127.1.170.131" port: 44771 } }
I20260923 20:08:53.788576 3336 raft_consensus.cc:348] T 47ebc61a3bfb40e8b7b6d350c957347d P 42013bee84ed41b59532dc3b7fd90adb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b56de26098334552bb9ae268d771c311" member_type: VOTER last_known_addr { host: "127.1.170.130" port: 33471 } } peers { permanent_uuid: "f1669496db8e44798ae7354b2d1f315b" member_type: VOTER last_known_addr { host: "127.1.170.129" port: 38261 } } peers { permanent_uuid: "42013bee84ed41b59532dc3b7fd90adb" member_type: VOTER last_known_addr { host: "127.1.170.131" port: 44771 } }
I20260923 20:08:53.789045 3335 raft_consensus.cc:374] T 47ebc61a3bfb40e8b7b6d350c957347d P f1669496db8e44798ae7354b2d1f315b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:08:53.789045 3336 raft_consensus.cc:374] T 47ebc61a3bfb40e8b7b6d350c957347d P 42013bee84ed41b59532dc3b7fd90adb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:08:53.789152 3336 raft_consensus.cc:729] T 47ebc61a3bfb40e8b7b6d350c957347d P 42013bee84ed41b59532dc3b7fd90adb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 42013bee84ed41b59532dc3b7fd90adb, State: Initialized, Role: FOLLOWER
I20260923 20:08:53.789152 3335 raft_consensus.cc:729] T 47ebc61a3bfb40e8b7b6d350c957347d P f1669496db8e44798ae7354b2d1f315b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1669496db8e44798ae7354b2d1f315b, State: Initialized, Role: FOLLOWER
I20260923 20:08:53.789805 3336 consensus_queue.cc:260] T 47ebc61a3bfb40e8b7b6d350c957347d P 42013bee84ed41b59532dc3b7fd90adb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b56de26098334552bb9ae268d771c311" member_type: VOTER last_known_addr { host: "127.1.170.130" port: 33471 } } peers { permanent_uuid: "f1669496db8e44798ae7354b2d1f315b" member_type: VOTER last_known_addr { host: "127.1.170.129" port: 38261 } } peers { permanent_uuid: "42013bee84ed41b59532dc3b7fd90adb" member_type: VOTER last_known_addr { host: "127.1.170.131" port: 44771 } }
I20260923 20:08:53.790110 3335 consensus_queue.cc:260] T 47ebc61a3bfb40e8b7b6d350c957347d P f1669496db8e44798ae7354b2d1f315b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b56de26098334552bb9ae268d771c311" member_type: VOTER last_known_addr { host: "127.1.170.130" port: 33471 } } peers { permanent_uuid: "f1669496db8e44798ae7354b2d1f315b" member_type: VOTER last_known_addr { host: "127.1.170.129" port: 38261 } } peers { permanent_uuid: "42013bee84ed41b59532dc3b7fd90adb" member_type: VOTER last_known_addr { host: "127.1.170.131" port: 44771 } }
I20260923 20:08:53.790684 3337 ts_tablet_manager.cc:1428] T 47ebc61a3bfb40e8b7b6d350c957347d P b56de26098334552bb9ae268d771c311: Time spent starting tablet: real 0.014s user 0.015s sys 0.000s
I20260923 20:08:53.792043 3316 heartbeater.cc:503] Master 127.1.170.190:39075 was elected leader, sending a full tablet report...
I20260923 20:08:53.792378 3336 ts_tablet_manager.cc:1428] T 47ebc61a3bfb40e8b7b6d350c957347d P 42013bee84ed41b59532dc3b7fd90adb: Time spent starting tablet: real 0.016s user 0.017s sys 0.000s
I20260923 20:08:53.792495 3335 ts_tablet_manager.cc:1428] T 47ebc61a3bfb40e8b7b6d350c957347d P f1669496db8e44798ae7354b2d1f315b: Time spent starting tablet: real 0.016s user 0.009s sys 0.008s
W20260923 20:08:53.810256 3039 tablet.cc:2369] T 47ebc61a3bfb40e8b7b6d350c957347d P f1669496db8e44798ae7354b2d1f315b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260923 20:08:53.826581 3178 tablet.cc:2369] T 47ebc61a3bfb40e8b7b6d350c957347d P b56de26098334552bb9ae268d771c311: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 20:08:53.878736 3343 raft_consensus.cc:482] T 47ebc61a3bfb40e8b7b6d350c957347d P f1669496db8e44798ae7354b2d1f315b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 20:08:53.879398 3343 raft_consensus.cc:504] T 47ebc61a3bfb40e8b7b6d350c957347d P f1669496db8e44798ae7354b2d1f315b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b56de26098334552bb9ae268d771c311" member_type: VOTER last_known_addr { host: "127.1.170.130" port: 33471 } } peers { permanent_uuid: "f1669496db8e44798ae7354b2d1f315b" member_type: VOTER last_known_addr { host: "127.1.170.129" port: 38261 } } peers { permanent_uuid: "42013bee84ed41b59532dc3b7fd90adb" member_type: VOTER last_known_addr { host: "127.1.170.131" port: 44771 } }
I20260923 20:08:53.881137 3343 leader_election.cc:288] T 47ebc61a3bfb40e8b7b6d350c957347d P f1669496db8e44798ae7354b2d1f315b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b56de26098334552bb9ae268d771c311 (127.1.170.130:33471), 42013bee84ed41b59532dc3b7fd90adb (127.1.170.131:44771)
I20260923 20:08:53.888285 3331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:08:53.881705 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.170.129:50147 (local address 127.1.170.130:33471)
0923 20:08:53.881948 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0923 20:08:53.881957 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 20:08:53.882547 (+ 590us) server_negotiation.cc:407] Connection header received
0923 20:08:53.882765 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:08:53.882771 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:08:53.882908 (+ 137us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:08:53.883024 (+ 116us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:08:53.883697 (+ 673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:08:53.884493 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:08:53.886476 (+ 1983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:08:53.887330 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:08:53.887498 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:08:53.887598 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0923 20:08:53.887886 (+ 288us) server_negotiation.cc:299] Negotiation successful
0923 20:08:53.888020 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260923 20:08:53.888314 3344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:08:53.881715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.130:33471 (local address 127.1.170.129:50147)
0923 20:08:53.882424 (+ 709us) negotiation.cc:107] Waiting for socket to connect
0923 20:08:53.882446 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0923 20:08:53.882593 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:08:53.883156 (+ 563us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:08:53.883162 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:08:53.883195 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:08:53.883543 (+ 348us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:08:53.883552 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:08:53.884626 (+ 1074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:08:53.884634 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:08:53.886304 (+ 1670us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:08:53.886316 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:08:53.887531 (+ 1215us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:08:53.887537 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:08:53.887716 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:08:53.887751 (+ 35us) client_negotiation.cc:769] Sending connection context
0923 20:08:53.887840 (+ 89us) client_negotiation.cc:240] Negotiation successful
0923 20:08:53.887977 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":562,"thread_start_us":400,"threads_started":1}
I20260923 20:08:53.889468 3334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:08:53.882825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.170.129:36729 (local address 127.1.170.131:44771)
0923 20:08:53.883058 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0923 20:08:53.883064 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:08:53.883354 (+ 290us) server_negotiation.cc:407] Connection header received
0923 20:08:53.883553 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:08:53.883558 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:08:53.883632 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:08:53.883730 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:08:53.884322 (+ 592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:08:53.885094 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:08:53.887258 (+ 2164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:08:53.888422 (+ 1164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:08:53.888582 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:08:53.888646 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0923 20:08:53.889088 (+ 442us) server_negotiation.cc:299] Negotiation successful
0923 20:08:53.889225 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260923 20:08:53.889685 3127 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47ebc61a3bfb40e8b7b6d350c957347d" candidate_uuid: "f1669496db8e44798ae7354b2d1f315b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b56de26098334552bb9ae268d771c311" is_pre_election: true
I20260923 20:08:53.890136 3345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:08:53.882547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.131:44771 (local address 127.1.170.129:36729)
0923 20:08:53.883243 (+ 696us) negotiation.cc:107] Waiting for socket to connect
0923 20:08:53.883260 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0923 20:08:53.883418 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:08:53.883755 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:08:53.883760 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:08:53.883782 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:08:53.884095 (+ 313us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:08:53.884102 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:08:53.885224 (+ 1122us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:08:53.885232 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:08:53.887085 (+ 1853us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:08:53.887099 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:08:53.888743 (+ 1644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:08:53.888750 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:08:53.888915 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:08:53.888947 (+ 32us) client_negotiation.cc:769] Sending connection context
0923 20:08:53.889637 (+ 690us) client_negotiation.cc:240] Negotiation successful
0923 20:08:53.889794 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":595,"thread_start_us":410,"threads_started":1}
I20260923 20:08:53.890317 3127 raft_consensus.cc:2454] T 47ebc61a3bfb40e8b7b6d350c957347d P b56de26098334552bb9ae268d771c311 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f1669496db8e44798ae7354b2d1f315b in term 0.
I20260923 20:08:53.891072 2921 leader_election.cc:302] T 47ebc61a3bfb40e8b7b6d350c957347d P f1669496db8e44798ae7354b2d1f315b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b56de26098334552bb9ae268d771c311, f1669496db8e44798ae7354b2d1f315b; no voters:
I20260923 20:08:53.891090 3270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47ebc61a3bfb40e8b7b6d350c957347d" candidate_uuid: "f1669496db8e44798ae7354b2d1f315b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "42013bee84ed41b59532dc3b7fd90adb" is_pre_election: true
I20260923 20:08:53.891609 3270 raft_consensus.cc:2454] T 47ebc61a3bfb40e8b7b6d350c957347d P 42013bee84ed41b59532dc3b7fd90adb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f1669496db8e44798ae7354b2d1f315b in term 0.
W20260923 20:08:53.891760 3317 tablet.cc:2369] T 47ebc61a3bfb40e8b7b6d350c957347d P 42013bee84ed41b59532dc3b7fd90adb: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 20:08:53.892127 3343 raft_consensus.cc:2781] T 47ebc61a3bfb40e8b7b6d350c957347d P f1669496db8e44798ae7354b2d1f315b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 20:08:53.892297 3343 raft_consensus.cc:482] T 47ebc61a3bfb40e8b7b6d350c957347d P f1669496db8e44798ae7354b2d1f315b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 20:08:53.892403 3343 raft_consensus.cc:3037] T 47ebc61a3bfb40e8b7b6d350c957347d P f1669496db8e44798ae7354b2d1f315b [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:08:53.894774 3343 raft_consensus.cc:504] T 47ebc61a3bfb40e8b7b6d350c957347d P f1669496db8e44798ae7354b2d1f315b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b56de26098334552bb9ae268d771c311" member_type: VOTER last_known_addr { host: "127.1.170.130" port: 33471 } } peers { permanent_uuid: "f1669496db8e44798ae7354b2d1f315b" member_type: VOTER last_known_addr { host: "127.1.170.129" port: 38261 } } peers { permanent_uuid: "42013bee84ed41b59532dc3b7fd90adb" member_type: VOTER last_known_addr { host: "127.1.170.131" port: 44771 } }
I20260923 20:08:53.895411 3343 leader_election.cc:288] T 47ebc61a3bfb40e8b7b6d350c957347d P f1669496db8e44798ae7354b2d1f315b [CANDIDATE]: Term 1 election: Requested vote from peers b56de26098334552bb9ae268d771c311 (127.1.170.130:33471), 42013bee84ed41b59532dc3b7fd90adb (127.1.170.131:44771)
I20260923 20:08:53.896086 3270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47ebc61a3bfb40e8b7b6d350c957347d" candidate_uuid: "f1669496db8e44798ae7354b2d1f315b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "42013bee84ed41b59532dc3b7fd90adb"
I20260923 20:08:53.895939 3127 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47ebc61a3bfb40e8b7b6d350c957347d" candidate_uuid: "f1669496db8e44798ae7354b2d1f315b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b56de26098334552bb9ae268d771c311"
I20260923 20:08:53.896306 3270 raft_consensus.cc:3037] T 47ebc61a3bfb40e8b7b6d350c957347d P 42013bee84ed41b59532dc3b7fd90adb [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:08:53.896448 3127 raft_consensus.cc:3037] T 47ebc61a3bfb40e8b7b6d350c957347d P b56de26098334552bb9ae268d771c311 [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:08:53.899310 3127 raft_consensus.cc:2454] T 47ebc61a3bfb40e8b7b6d350c957347d P b56de26098334552bb9ae268d771c311 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f1669496db8e44798ae7354b2d1f315b in term 1.
I20260923 20:08:53.899628 3270 raft_consensus.cc:2454] T 47ebc61a3bfb40e8b7b6d350c957347d P 42013bee84ed41b59532dc3b7fd90adb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f1669496db8e44798ae7354b2d1f315b in term 1.
I20260923 20:08:53.900187 2921 leader_election.cc:302] T 47ebc61a3bfb40e8b7b6d350c957347d P f1669496db8e44798ae7354b2d1f315b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b56de26098334552bb9ae268d771c311, f1669496db8e44798ae7354b2d1f315b; no voters:
I20260923 20:08:53.900687 3343 raft_consensus.cc:2781] T 47ebc61a3bfb40e8b7b6d350c957347d P f1669496db8e44798ae7354b2d1f315b [term 1 FOLLOWER]: Leader election won for term 1
I20260923 20:08:53.901593 3343 raft_consensus.cc:686] T 47ebc61a3bfb40e8b7b6d350c957347d P f1669496db8e44798ae7354b2d1f315b [term 1 LEADER]: Becoming Leader. State: Replica: f1669496db8e44798ae7354b2d1f315b, State: Running, Role: LEADER
I20260923 20:08:53.902253 3343 consensus_queue.cc:237] T 47ebc61a3bfb40e8b7b6d350c957347d P f1669496db8e44798ae7354b2d1f315b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b56de26098334552bb9ae268d771c311" member_type: VOTER last_known_addr { host: "127.1.170.130" port: 33471 } } peers { permanent_uuid: "f1669496db8e44798ae7354b2d1f315b" member_type: VOTER last_known_addr { host: "127.1.170.129" port: 38261 } } peers { permanent_uuid: "42013bee84ed41b59532dc3b7fd90adb" member_type: VOTER last_known_addr { host: "127.1.170.131" port: 44771 } }
I20260923 20:08:53.907394 2829 catalog_manager.cc:5495] T 47ebc61a3bfb40e8b7b6d350c957347d P f1669496db8e44798ae7354b2d1f315b reported cstate change: term changed from 0 to 1, leader changed from <none> to f1669496db8e44798ae7354b2d1f315b (127.1.170.129). New cstate: current_term: 1 leader_uuid: "f1669496db8e44798ae7354b2d1f315b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b56de26098334552bb9ae268d771c311" member_type: VOTER last_known_addr { host: "127.1.170.130" port: 33471 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f1669496db8e44798ae7354b2d1f315b" member_type: VOTER last_known_addr { host: "127.1.170.129" port: 38261 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "42013bee84ed41b59532dc3b7fd90adb" member_type: VOTER last_known_addr { host: "127.1.170.131" port: 44771 } health_report { overall_health: UNKNOWN } } }
I20260923 20:08:53.954784 3332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:08:53.947354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52276 (local address 127.1.170.129:38261)
0923 20:08:53.947580 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0923 20:08:53.947588 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 20:08:53.947620 (+ 32us) server_negotiation.cc:407] Connection header received
0923 20:08:53.947972 (+ 352us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:08:53.947976 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:08:53.948070 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:08:53.948194 (+ 124us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 20:08:53.948609 (+ 415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:08:53.949459 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:08:53.951121 (+ 1662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:08:53.951525 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:08:53.951636 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:08:53.952776 (+ 1140us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 20:08:53.953822 (+ 1046us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 20:08:53.954091 (+ 269us) server_negotiation.cc:1035] Waiting for connection context
0923 20:08:53.954343 (+ 252us) server_negotiation.cc:299] Negotiation successful
0923 20:08:53.954479 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260923 20:08:53.966115 3270 raft_consensus.cc:1261] T 47ebc61a3bfb40e8b7b6d350c957347d P 42013bee84ed41b59532dc3b7fd90adb [term 1 FOLLOWER]: Refusing update from remote peer f1669496db8e44798ae7354b2d1f315b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 20:08:53.967456 3343 consensus_queue.cc:1035] T 47ebc61a3bfb40e8b7b6d350c957347d P f1669496db8e44798ae7354b2d1f315b [LEADER]: Connected to new peer: Peer: permanent_uuid: "42013bee84ed41b59532dc3b7fd90adb" member_type: VOTER last_known_addr { host: "127.1.170.131" port: 44771 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 20:08:53.968170 3127 raft_consensus.cc:1261] T 47ebc61a3bfb40e8b7b6d350c957347d P b56de26098334552bb9ae268d771c311 [term 1 FOLLOWER]: Refusing update from remote peer f1669496db8e44798ae7354b2d1f315b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 20:08:53.969022 3343 consensus_queue.cc:1035] T 47ebc61a3bfb40e8b7b6d350c957347d P f1669496db8e44798ae7354b2d1f315b [LEADER]: Connected to new peer: Peer: permanent_uuid: "b56de26098334552bb9ae268d771c311" member_type: VOTER last_known_addr { host: "127.1.170.130" port: 33471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 20:08:53.987183 3351 mvcc.cc:205] Tried to move back new op lower bound from 7332635172710080512 to 7332635172468035584. Current Snapshot: MvccSnapshot[applied={T|T < 7332635172710080512}]
I20260923 20:08:54.007993 2831 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42432:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 20:08:54.008594 2831 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42432:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 20:08:54.014060 2831 catalog_manager.cc:5782] T 00000000000000000000000000000000 P be44f549a3ba42f48a99b54e16576e8a: Sending DeleteTablet for 3 replicas of tablet 47ebc61a3bfb40e8b7b6d350c957347d
I20260923 20:08:54.015538 2968 tablet_service.cc:1514] Processing DeleteTablet for tablet 47ebc61a3bfb40e8b7b6d350c957347d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 20:08:54 UTC) from {username='kudu', principal='kudu/127.1.170.190@KRBTEST.COM'} at 127.0.0.1:52274
I20260923 20:08:54.015584 3246 tablet_service.cc:1514] Processing DeleteTablet for tablet 47ebc61a3bfb40e8b7b6d350c957347d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 20:08:54 UTC) from {username='kudu', principal='kudu/127.1.170.190@KRBTEST.COM'} at 127.0.0.1:35654
I20260923 20:08:54.016559 3107 tablet_service.cc:1514] Processing DeleteTablet for tablet 47ebc61a3bfb40e8b7b6d350c957347d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 20:08:54 UTC) from {username='kudu', principal='kudu/127.1.170.190@KRBTEST.COM'} at 127.0.0.1:51168
I20260923 20:08:54.017617 2829 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42432:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 20:08:54.018577 2829 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 20:08:54.020710 3362 tablet_replica.cc:330] T 47ebc61a3bfb40e8b7b6d350c957347d P b56de26098334552bb9ae268d771c311: stopping tablet replica
I20260923 20:08:54.027248 3362 raft_consensus.cc:2229] T 47ebc61a3bfb40e8b7b6d350c957347d P b56de26098334552bb9ae268d771c311 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 20:08:54.027338 2967 tablet_service.cc:1467] Processing CreateTablet for tablet cdabeae2e5284ae9a299e9ad611006b6 (DEFAULT_TABLE table=test-table [id=fba67baa5f5045058d8917ff2c0d69f3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 20:08:54.027827 3362 raft_consensus.cc:2258] T 47ebc61a3bfb40e8b7b6d350c957347d P b56de26098334552bb9ae268d771c311 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 20:08:54.027951 2967 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cdabeae2e5284ae9a299e9ad611006b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:08:54.027881 3107 tablet_service.cc:1467] Processing CreateTablet for tablet cdabeae2e5284ae9a299e9ad611006b6 (DEFAULT_TABLE table=test-table [id=fba67baa5f5045058d8917ff2c0d69f3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 20:08:54.028309 3107 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cdabeae2e5284ae9a299e9ad611006b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:08:54.030393 3245 tablet_service.cc:1467] Processing CreateTablet for tablet cdabeae2e5284ae9a299e9ad611006b6 (DEFAULT_TABLE table=test-table [id=fba67baa5f5045058d8917ff2c0d69f3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 20:08:54.030915 3245 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cdabeae2e5284ae9a299e9ad611006b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:08:54.032567 3362 ts_tablet_manager.cc:1905] T 47ebc61a3bfb40e8b7b6d350c957347d P b56de26098334552bb9ae268d771c311: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 20:08:54.041505 3337 tablet_bootstrap.cc:492] T cdabeae2e5284ae9a299e9ad611006b6 P b56de26098334552bb9ae268d771c311: Bootstrap starting.
I20260923 20:08:54.041646 3360 tablet_replica.cc:330] T 47ebc61a3bfb40e8b7b6d350c957347d P 42013bee84ed41b59532dc3b7fd90adb: stopping tablet replica
I20260923 20:08:54.042201 3360 raft_consensus.cc:2229] T 47ebc61a3bfb40e8b7b6d350c957347d P 42013bee84ed41b59532dc3b7fd90adb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 20:08:54.042760 3360 raft_consensus.cc:2258] T 47ebc61a3bfb40e8b7b6d350c957347d P 42013bee84ed41b59532dc3b7fd90adb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 20:08:54.043292 3337 tablet_bootstrap.cc:654] T cdabeae2e5284ae9a299e9ad611006b6 P b56de26098334552bb9ae268d771c311: Neither blocks nor log segments found. Creating new log.
I20260923 20:08:54.043602 3361 tablet_replica.cc:330] T 47ebc61a3bfb40e8b7b6d350c957347d P f1669496db8e44798ae7354b2d1f315b: stopping tablet replica
I20260923 20:08:54.044082 3361 raft_consensus.cc:2229] T 47ebc61a3bfb40e8b7b6d350c957347d P f1669496db8e44798ae7354b2d1f315b [term 1 LEADER]: Raft consensus shutting down.
I20260923 20:08:54.045770 3336 tablet_bootstrap.cc:492] T cdabeae2e5284ae9a299e9ad611006b6 P 42013bee84ed41b59532dc3b7fd90adb: Bootstrap starting.
I20260923 20:08:54.046826 3361 raft_consensus.cc:2258] T 47ebc61a3bfb40e8b7b6d350c957347d P f1669496db8e44798ae7354b2d1f315b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 20:08:54.047487 3336 tablet_bootstrap.cc:654] T cdabeae2e5284ae9a299e9ad611006b6 P 42013bee84ed41b59532dc3b7fd90adb: Neither blocks nor log segments found. Creating new log.
I20260923 20:08:54.048485 3335 tablet_bootstrap.cc:492] T cdabeae2e5284ae9a299e9ad611006b6 P f1669496db8e44798ae7354b2d1f315b: Bootstrap starting.
I20260923 20:08:54.050326 3335 tablet_bootstrap.cc:654] T cdabeae2e5284ae9a299e9ad611006b6 P f1669496db8e44798ae7354b2d1f315b: Neither blocks nor log segments found. Creating new log.
I20260923 20:08:54.051095 3361 ts_tablet_manager.cc:1905] T 47ebc61a3bfb40e8b7b6d350c957347d P f1669496db8e44798ae7354b2d1f315b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 20:08:54.055675 3335 tablet_bootstrap.cc:492] T cdabeae2e5284ae9a299e9ad611006b6 P f1669496db8e44798ae7354b2d1f315b: No bootstrap required, opened a new log
I20260923 20:08:54.055861 3335 ts_tablet_manager.cc:1397] T cdabeae2e5284ae9a299e9ad611006b6 P f1669496db8e44798ae7354b2d1f315b: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.000s
I20260923 20:08:54.056764 3335 raft_consensus.cc:348] T cdabeae2e5284ae9a299e9ad611006b6 P f1669496db8e44798ae7354b2d1f315b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1669496db8e44798ae7354b2d1f315b" member_type: VOTER last_known_addr { host: "127.1.170.129" port: 38261 } } peers { permanent_uuid: "b56de26098334552bb9ae268d771c311" member_type: VOTER last_known_addr { host: "127.1.170.130" port: 33471 } } peers { permanent_uuid: "42013bee84ed41b59532dc3b7fd90adb" member_type: VOTER last_known_addr { host: "127.1.170.131" port: 44771 } }
I20260923 20:08:54.057024 3335 raft_consensus.cc:374] T cdabeae2e5284ae9a299e9ad611006b6 P f1669496db8e44798ae7354b2d1f315b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:08:54.057123 3335 raft_consensus.cc:729] T cdabeae2e5284ae9a299e9ad611006b6 P f1669496db8e44798ae7354b2d1f315b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1669496db8e44798ae7354b2d1f315b, State: Initialized, Role: FOLLOWER
I20260923 20:08:54.057425 3335 consensus_queue.cc:260] T cdabeae2e5284ae9a299e9ad611006b6 P f1669496db8e44798ae7354b2d1f315b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1669496db8e44798ae7354b2d1f315b" member_type: VOTER last_known_addr { host: "127.1.170.129" port: 38261 } } peers { permanent_uuid: "b56de26098334552bb9ae268d771c311" member_type: VOTER last_known_addr { host: "127.1.170.130" port: 33471 } } peers { permanent_uuid: "42013bee84ed41b59532dc3b7fd90adb" member_type: VOTER last_known_addr { host: "127.1.170.131" port: 44771 } }
I20260923 20:08:54.058681 3362 ts_tablet_manager.cc:1918] T 47ebc61a3bfb40e8b7b6d350c957347d P b56de26098334552bb9ae268d771c311: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 20:08:54.058811 3335 ts_tablet_manager.cc:1428] T cdabeae2e5284ae9a299e9ad611006b6 P f1669496db8e44798ae7354b2d1f315b: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260923 20:08:54.058873 3362 log.cc:1196] T 47ebc61a3bfb40e8b7b6d350c957347d P b56de26098334552bb9ae268d771c311: Deleting WAL directory at /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790194121577027-1706-0/minicluster-data/ts-1/wal/wals/47ebc61a3bfb40e8b7b6d350c957347d
I20260923 20:08:54.059661 3362 ts_tablet_manager.cc:1939] T 47ebc61a3bfb40e8b7b6d350c957347d P b56de26098334552bb9ae268d771c311: Deleting consensus metadata
I20260923 20:08:54.062060 2818 catalog_manager.cc:4841] TS b56de26098334552bb9ae268d771c311 (127.1.170.130:33471): tablet 47ebc61a3bfb40e8b7b6d350c957347d (table test-table [id=d14c6f0609514b4da971b482e687b415]) successfully deleted
I20260923 20:08:54.063334 3336 tablet_bootstrap.cc:492] T cdabeae2e5284ae9a299e9ad611006b6 P 42013bee84ed41b59532dc3b7fd90adb: No bootstrap required, opened a new log
I20260923 20:08:54.063565 3336 ts_tablet_manager.cc:1397] T cdabeae2e5284ae9a299e9ad611006b6 P 42013bee84ed41b59532dc3b7fd90adb: Time spent bootstrapping tablet: real 0.018s user 0.001s sys 0.008s
I20260923 20:08:54.064551 3336 raft_consensus.cc:348] T cdabeae2e5284ae9a299e9ad611006b6 P 42013bee84ed41b59532dc3b7fd90adb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1669496db8e44798ae7354b2d1f315b" member_type: VOTER last_known_addr { host: "127.1.170.129" port: 38261 } } peers { permanent_uuid: "b56de26098334552bb9ae268d771c311" member_type: VOTER last_known_addr { host: "127.1.170.130" port: 33471 } } peers { permanent_uuid: "42013bee84ed41b59532dc3b7fd90adb" member_type: VOTER last_known_addr { host: "127.1.170.131" port: 44771 } }
I20260923 20:08:54.064937 3336 raft_consensus.cc:374] T cdabeae2e5284ae9a299e9ad611006b6 P 42013bee84ed41b59532dc3b7fd90adb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:08:54.065080 3336 raft_consensus.cc:729] T cdabeae2e5284ae9a299e9ad611006b6 P 42013bee84ed41b59532dc3b7fd90adb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 42013bee84ed41b59532dc3b7fd90adb, State: Initialized, Role: FOLLOWER
I20260923 20:08:54.065444 3336 consensus_queue.cc:260] T cdabeae2e5284ae9a299e9ad611006b6 P 42013bee84ed41b59532dc3b7fd90adb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1669496db8e44798ae7354b2d1f315b" member_type: VOTER last_known_addr { host: "127.1.170.129" port: 38261 } } peers { permanent_uuid: "b56de26098334552bb9ae268d771c311" member_type: VOTER last_known_addr { host: "127.1.170.130" port: 33471 } } peers { permanent_uuid: "42013bee84ed41b59532dc3b7fd90adb" member_type: VOTER last_known_addr { host: "127.1.170.131" port: 44771 } }
I20260923 20:08:54.067895 3361 ts_tablet_manager.cc:1918] T 47ebc61a3bfb40e8b7b6d350c957347d P f1669496db8e44798ae7354b2d1f315b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 20:08:54.068145 3361 log.cc:1196] T 47ebc61a3bfb40e8b7b6d350c957347d P f1669496db8e44798ae7354b2d1f315b: Deleting WAL directory at /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790194121577027-1706-0/minicluster-data/ts-0/wal/wals/47ebc61a3bfb40e8b7b6d350c957347d
I20260923 20:08:54.068594 3336 ts_tablet_manager.cc:1428] T cdabeae2e5284ae9a299e9ad611006b6 P 42013bee84ed41b59532dc3b7fd90adb: Time spent starting tablet: real 0.005s user 0.001s sys 0.000s
I20260923 20:08:54.068732 3361 ts_tablet_manager.cc:1939] T 47ebc61a3bfb40e8b7b6d350c957347d P f1669496db8e44798ae7354b2d1f315b: Deleting consensus metadata
I20260923 20:08:54.070899 2815 catalog_manager.cc:4841] TS f1669496db8e44798ae7354b2d1f315b (127.1.170.129:38261): tablet 47ebc61a3bfb40e8b7b6d350c957347d (table test-table [id=d14c6f0609514b4da971b482e687b415]) successfully deleted
I20260923 20:08:54.072825 3360 ts_tablet_manager.cc:1905] T 47ebc61a3bfb40e8b7b6d350c957347d P 42013bee84ed41b59532dc3b7fd90adb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 20:08:54.073137 3337 tablet_bootstrap.cc:492] T cdabeae2e5284ae9a299e9ad611006b6 P b56de26098334552bb9ae268d771c311: No bootstrap required, opened a new log
I20260923 20:08:54.073433 3337 ts_tablet_manager.cc:1397] T cdabeae2e5284ae9a299e9ad611006b6 P b56de26098334552bb9ae268d771c311: Time spent bootstrapping tablet: real 0.032s user 0.008s sys 0.011s
I20260923 20:08:54.074383 3337 raft_consensus.cc:348] T cdabeae2e5284ae9a299e9ad611006b6 P b56de26098334552bb9ae268d771c311 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1669496db8e44798ae7354b2d1f315b" member_type: VOTER last_known_addr { host: "127.1.170.129" port: 38261 } } peers { permanent_uuid: "b56de26098334552bb9ae268d771c311" member_type: VOTER last_known_addr { host: "127.1.170.130" port: 33471 } } peers { permanent_uuid: "42013bee84ed41b59532dc3b7fd90adb" member_type: VOTER last_known_addr { host: "127.1.170.131" port: 44771 } }
I20260923 20:08:54.074699 3337 raft_consensus.cc:374] T cdabeae2e5284ae9a299e9ad611006b6 P b56de26098334552bb9ae268d771c311 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:08:54.074842 3337 raft_consensus.cc:729] T cdabeae2e5284ae9a299e9ad611006b6 P b56de26098334552bb9ae268d771c311 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b56de26098334552bb9ae268d771c311, State: Initialized, Role: FOLLOWER
I20260923 20:08:54.075250 3337 consensus_queue.cc:260] T cdabeae2e5284ae9a299e9ad611006b6 P b56de26098334552bb9ae268d771c311 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1669496db8e44798ae7354b2d1f315b" member_type: VOTER last_known_addr { host: "127.1.170.129" port: 38261 } } peers { permanent_uuid: "b56de26098334552bb9ae268d771c311" member_type: VOTER last_known_addr { host: "127.1.170.130" port: 33471 } } peers { permanent_uuid: "42013bee84ed41b59532dc3b7fd90adb" member_type: VOTER last_known_addr { host: "127.1.170.131" port: 44771 } }
I20260923 20:08:54.076000 3337 ts_tablet_manager.cc:1428] T cdabeae2e5284ae9a299e9ad611006b6 P b56de26098334552bb9ae268d771c311: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260923 20:08:54.080618 3360 ts_tablet_manager.cc:1918] T 47ebc61a3bfb40e8b7b6d350c957347d P 42013bee84ed41b59532dc3b7fd90adb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 20:08:54.080812 3360 log.cc:1196] T 47ebc61a3bfb40e8b7b6d350c957347d P 42013bee84ed41b59532dc3b7fd90adb: Deleting WAL directory at /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790194121577027-1706-0/minicluster-data/ts-2/wal/wals/47ebc61a3bfb40e8b7b6d350c957347d
I20260923 20:08:54.081332 3360 ts_tablet_manager.cc:1939] T 47ebc61a3bfb40e8b7b6d350c957347d P 42013bee84ed41b59532dc3b7fd90adb: Deleting consensus metadata
I20260923 20:08:54.083150 2818 catalog_manager.cc:4841] TS 42013bee84ed41b59532dc3b7fd90adb (127.1.170.131:44771): tablet 47ebc61a3bfb40e8b7b6d350c957347d (table test-table [id=d14c6f0609514b4da971b482e687b415]) successfully deleted
I20260923 20:08:54.245188 3341 raft_consensus.cc:482] T cdabeae2e5284ae9a299e9ad611006b6 P b56de26098334552bb9ae268d771c311 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 20:08:54.245435 3341 raft_consensus.cc:504] T cdabeae2e5284ae9a299e9ad611006b6 P b56de26098334552bb9ae268d771c311 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1669496db8e44798ae7354b2d1f315b" member_type: VOTER last_known_addr { host: "127.1.170.129" port: 38261 } } peers { permanent_uuid: "b56de26098334552bb9ae268d771c311" member_type: VOTER last_known_addr { host: "127.1.170.130" port: 33471 } } peers { permanent_uuid: "42013bee84ed41b59532dc3b7fd90adb" member_type: VOTER last_known_addr { host: "127.1.170.131" port: 44771 } }
I20260923 20:08:54.247346 3341 leader_election.cc:288] T cdabeae2e5284ae9a299e9ad611006b6 P b56de26098334552bb9ae268d771c311 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f1669496db8e44798ae7354b2d1f315b (127.1.170.129:38261), 42013bee84ed41b59532dc3b7fd90adb (127.1.170.131:44771)
I20260923 20:08:54.255843 3332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:08:54.248188 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.170.130:56821 (local address 127.1.170.129:38261)
0923 20:08:54.248369 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0923 20:08:54.248375 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:08:54.248697 (+ 322us) server_negotiation.cc:407] Connection header received
0923 20:08:54.249007 (+ 310us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:08:54.249013 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:08:54.249099 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:08:54.249153 (+ 54us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:08:54.250319 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:08:54.251643 (+ 1324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:08:54.253541 (+ 1898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:08:54.254726 (+ 1185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:08:54.254898 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:08:54.254975 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0923 20:08:54.255346 (+ 371us) server_negotiation.cc:299] Negotiation successful
0923 20:08:54.255475 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260923 20:08:54.255852 3366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:08:54.247710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.129:38261 (local address 127.1.170.130:56821)
0923 20:08:54.248557 (+ 847us) negotiation.cc:107] Waiting for socket to connect
0923 20:08:54.248583 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0923 20:08:54.248785 (+ 202us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:08:54.249377 (+ 592us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:08:54.249382 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:08:54.249536 (+ 154us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:08:54.250112 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:08:54.250131 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:08:54.251811 (+ 1680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:08:54.251814 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:08:54.253366 (+ 1552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:08:54.253379 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:08:54.254950 (+ 1571us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:08:54.254957 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:08:54.255149 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:08:54.255186 (+ 37us) client_negotiation.cc:769] Sending connection context
0923 20:08:54.255302 (+ 116us) client_negotiation.cc:240] Negotiation successful
0923 20:08:54.255456 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":619,"thread_start_us":729,"threads_started":1}
I20260923 20:08:54.256461 3334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:08:54.247787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.170.130:57529 (local address 127.1.170.131:44771)
0923 20:08:54.247990 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0923 20:08:54.247997 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:08:54.249270 (+ 1273us) server_negotiation.cc:407] Connection header received
0923 20:08:54.249424 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:08:54.249429 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:08:54.249550 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:08:54.249666 (+ 116us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:08:54.250368 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:08:54.251620 (+ 1252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:08:54.254195 (+ 2575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:08:54.255284 (+ 1089us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:08:54.255431 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:08:54.255495 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0923 20:08:54.256072 (+ 577us) server_negotiation.cc:299] Negotiation successful
0923 20:08:54.256207 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260923 20:08:54.257145 3367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:08:54.247776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.131:44771 (local address 127.1.170.130:57529)
0923 20:08:54.249147 (+ 1371us) negotiation.cc:107] Waiting for socket to connect
0923 20:08:54.249165 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0923 20:08:54.249312 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:08:54.249666 (+ 354us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:08:54.249671 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:08:54.249699 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:08:54.250218 (+ 519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:08:54.250230 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:08:54.251779 (+ 1549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:08:54.251785 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:08:54.254010 (+ 2225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:08:54.254025 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:08:54.255670 (+ 1645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:08:54.255677 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:08:54.255874 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:08:54.255915 (+ 41us) client_negotiation.cc:769] Sending connection context
0923 20:08:54.256689 (+ 774us) client_negotiation.cc:240] Negotiation successful
0923 20:08:54.256830 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1255,"mutex_wait_us":4,"thread_start_us":1078,"threads_started":1}
I20260923 20:08:54.257336 2992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cdabeae2e5284ae9a299e9ad611006b6" candidate_uuid: "b56de26098334552bb9ae268d771c311" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1669496db8e44798ae7354b2d1f315b" is_pre_election: true
I20260923 20:08:54.257797 3270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cdabeae2e5284ae9a299e9ad611006b6" candidate_uuid: "b56de26098334552bb9ae268d771c311" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "42013bee84ed41b59532dc3b7fd90adb" is_pre_election: true
I20260923 20:08:54.257859 2992 raft_consensus.cc:2454] T cdabeae2e5284ae9a299e9ad611006b6 P f1669496db8e44798ae7354b2d1f315b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b56de26098334552bb9ae268d771c311 in term 0.
I20260923 20:08:54.258100 3270 raft_consensus.cc:2454] T cdabeae2e5284ae9a299e9ad611006b6 P 42013bee84ed41b59532dc3b7fd90adb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b56de26098334552bb9ae268d771c311 in term 0.
I20260923 20:08:54.258569 3058 leader_election.cc:302] T cdabeae2e5284ae9a299e9ad611006b6 P b56de26098334552bb9ae268d771c311 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 42013bee84ed41b59532dc3b7fd90adb, b56de26098334552bb9ae268d771c311, f1669496db8e44798ae7354b2d1f315b; no voters:
I20260923 20:08:54.259050 3341 raft_consensus.cc:2781] T cdabeae2e5284ae9a299e9ad611006b6 P b56de26098334552bb9ae268d771c311 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 20:08:54.259188 3341 raft_consensus.cc:482] T cdabeae2e5284ae9a299e9ad611006b6 P b56de26098334552bb9ae268d771c311 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 20:08:54.259282 3341 raft_consensus.cc:3037] T cdabeae2e5284ae9a299e9ad611006b6 P b56de26098334552bb9ae268d771c311 [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:08:54.261361 3341 raft_consensus.cc:504] T cdabeae2e5284ae9a299e9ad611006b6 P b56de26098334552bb9ae268d771c311 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1669496db8e44798ae7354b2d1f315b" member_type: VOTER last_known_addr { host: "127.1.170.129" port: 38261 } } peers { permanent_uuid: "b56de26098334552bb9ae268d771c311" member_type: VOTER last_known_addr { host: "127.1.170.130" port: 33471 } } peers { permanent_uuid: "42013bee84ed41b59532dc3b7fd90adb" member_type: VOTER last_known_addr { host: "127.1.170.131" port: 44771 } }
I20260923 20:08:54.262115 3341 leader_election.cc:288] T cdabeae2e5284ae9a299e9ad611006b6 P b56de26098334552bb9ae268d771c311 [CANDIDATE]: Term 1 election: Requested vote from peers f1669496db8e44798ae7354b2d1f315b (127.1.170.129:38261), 42013bee84ed41b59532dc3b7fd90adb (127.1.170.131:44771)
I20260923 20:08:54.262517 2992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cdabeae2e5284ae9a299e9ad611006b6" candidate_uuid: "b56de26098334552bb9ae268d771c311" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1669496db8e44798ae7354b2d1f315b"
I20260923 20:08:54.262749 2992 raft_consensus.cc:3037] T cdabeae2e5284ae9a299e9ad611006b6 P f1669496db8e44798ae7354b2d1f315b [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:08:54.262814 3270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cdabeae2e5284ae9a299e9ad611006b6" candidate_uuid: "b56de26098334552bb9ae268d771c311" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "42013bee84ed41b59532dc3b7fd90adb"
I20260923 20:08:54.263023 3270 raft_consensus.cc:3037] T cdabeae2e5284ae9a299e9ad611006b6 P 42013bee84ed41b59532dc3b7fd90adb [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:08:54.266309 3270 raft_consensus.cc:2454] T cdabeae2e5284ae9a299e9ad611006b6 P 42013bee84ed41b59532dc3b7fd90adb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b56de26098334552bb9ae268d771c311 in term 1.
I20260923 20:08:54.266386 2992 raft_consensus.cc:2454] T cdabeae2e5284ae9a299e9ad611006b6 P f1669496db8e44798ae7354b2d1f315b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b56de26098334552bb9ae268d771c311 in term 1.
I20260923 20:08:54.266891 3058 leader_election.cc:302] T cdabeae2e5284ae9a299e9ad611006b6 P b56de26098334552bb9ae268d771c311 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b56de26098334552bb9ae268d771c311, f1669496db8e44798ae7354b2d1f315b; no voters:
I20260923 20:08:54.267376 3341 raft_consensus.cc:2781] T cdabeae2e5284ae9a299e9ad611006b6 P b56de26098334552bb9ae268d771c311 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 20:08:54.267834 3341 raft_consensus.cc:686] T cdabeae2e5284ae9a299e9ad611006b6 P b56de26098334552bb9ae268d771c311 [term 1 LEADER]: Becoming Leader. State: Replica: b56de26098334552bb9ae268d771c311, State: Running, Role: LEADER
I20260923 20:08:54.268457 3341 consensus_queue.cc:237] T cdabeae2e5284ae9a299e9ad611006b6 P b56de26098334552bb9ae268d771c311 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1669496db8e44798ae7354b2d1f315b" member_type: VOTER last_known_addr { host: "127.1.170.129" port: 38261 } } peers { permanent_uuid: "b56de26098334552bb9ae268d771c311" member_type: VOTER last_known_addr { host: "127.1.170.130" port: 33471 } } peers { permanent_uuid: "42013bee84ed41b59532dc3b7fd90adb" member_type: VOTER last_known_addr { host: "127.1.170.131" port: 44771 } }
I20260923 20:08:54.271981 2829 catalog_manager.cc:5495] T cdabeae2e5284ae9a299e9ad611006b6 P b56de26098334552bb9ae268d771c311 reported cstate change: term changed from 0 to 1, leader changed from <none> to b56de26098334552bb9ae268d771c311 (127.1.170.130). New cstate: current_term: 1 leader_uuid: "b56de26098334552bb9ae268d771c311" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1669496db8e44798ae7354b2d1f315b" member_type: VOTER last_known_addr { host: "127.1.170.129" port: 38261 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b56de26098334552bb9ae268d771c311" member_type: VOTER last_known_addr { host: "127.1.170.130" port: 33471 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "42013bee84ed41b59532dc3b7fd90adb" member_type: VOTER last_known_addr { host: "127.1.170.131" port: 44771 } health_report { overall_health: UNKNOWN } } }
Sep 23 20:08:54 dist-test-slave-th4g krb5kdc[2786](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194131, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.170.190@KRBTEST.COM for kudu/127.1.170.190@KRBTEST.COM
I20260923 20:08:54.375916 3376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:08:54.356556 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.190:39075 (local address 127.0.0.1:42436)
0923 20:08:54.357326 (+ 770us) negotiation.cc:107] Waiting for socket to connect
0923 20:08:54.357351 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0923 20:08:54.357524 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:08:54.358323 (+ 799us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:08:54.358328 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:08:54.359645 (+ 1317us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:08:54.359982 (+ 337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:08:54.359999 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:08:54.361644 (+ 1645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:08:54.361651 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:08:54.363320 (+ 1669us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:08:54.363336 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:08:54.364133 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:08:54.364141 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:08:54.364333 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:08:54.365585 (+ 1252us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:08:54.365621 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:08:54.368445 (+ 2824us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:08:54.373055 (+ 4610us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:08:54.373068 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:08:54.373075 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:08:54.373550 (+ 475us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:08:54.374081 (+ 531us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:08:54.374084 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:08:54.374087 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:08:54.374243 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:08:54.374960 (+ 717us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:08:54.374966 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:08:54.375134 (+ 168us) client_negotiation.cc:769] Sending connection context
0923 20:08:54.375320 (+ 186us) client_negotiation.cc:240] Negotiation successful
0923 20:08:54.375538 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":581,"thread_start_us":375,"threads_started":1}
I20260923 20:08:54.376642 3375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:08:54.356584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42436 (local address 127.1.170.190:39075)
0923 20:08:54.357859 (+ 1275us) server_negotiation.cc:206] Beginning negotiation
0923 20:08:54.357866 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:08:54.357933 (+ 67us) server_negotiation.cc:407] Connection header received
0923 20:08:54.358000 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:08:54.358004 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:08:54.358065 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:08:54.358172 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:08:54.360313 (+ 2141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:08:54.361425 (+ 1112us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:08:54.363507 (+ 2082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:08:54.363931 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:08:54.364031 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:08:54.368682 (+ 4651us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:08:54.368691 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:08:54.368697 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:08:54.368735 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:08:54.372874 (+ 4139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:08:54.373708 (+ 834us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:08:54.373714 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:08:54.373717 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:08:54.373956 (+ 239us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:08:54.374366 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:08:54.374372 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:08:54.374375 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:08:54.374780 (+ 405us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:08:54.374936 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0923 20:08:54.375489 (+ 553us) server_negotiation.cc:299] Negotiation successful
0923 20:08:54.375720 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1117,"thread_start_us":881,"threads_started":1}
I20260923 20:08:54.382647 2829 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.1.170.190@KRBTEST.COM'} at 127.0.0.1:42436:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260923 20:08:54.384022 2829 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 20:08:54.392916 2967 tablet_service.cc:1467] Processing CreateTablet for tablet 0df49309d341414aaf775a5d9a29b2ec (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=fbbf8f5032024b89a6d3c68091ec0ce8]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260923 20:08:54.393519 2967 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0df49309d341414aaf775a5d9a29b2ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:08:54.394066 3245 tablet_service.cc:1467] Processing CreateTablet for tablet 0df49309d341414aaf775a5d9a29b2ec (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=fbbf8f5032024b89a6d3c68091ec0ce8]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260923 20:08:54.394558 3245 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0df49309d341414aaf775a5d9a29b2ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:08:54.398936 3335 tablet_bootstrap.cc:492] T 0df49309d341414aaf775a5d9a29b2ec P f1669496db8e44798ae7354b2d1f315b: Bootstrap starting.
I20260923 20:08:54.399595 3107 tablet_service.cc:1467] Processing CreateTablet for tablet 0df49309d341414aaf775a5d9a29b2ec (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=fbbf8f5032024b89a6d3c68091ec0ce8]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260923 20:08:54.400166 3107 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0df49309d341414aaf775a5d9a29b2ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:08:54.401254 3335 tablet_bootstrap.cc:654] T 0df49309d341414aaf775a5d9a29b2ec P f1669496db8e44798ae7354b2d1f315b: Neither blocks nor log segments found. Creating new log.
I20260923 20:08:54.402678 3336 tablet_bootstrap.cc:492] T 0df49309d341414aaf775a5d9a29b2ec P 42013bee84ed41b59532dc3b7fd90adb: Bootstrap starting.
I20260923 20:08:54.404327 3336 tablet_bootstrap.cc:654] T 0df49309d341414aaf775a5d9a29b2ec P 42013bee84ed41b59532dc3b7fd90adb: Neither blocks nor log segments found. Creating new log.
I20260923 20:08:54.411000 3337 tablet_bootstrap.cc:492] T 0df49309d341414aaf775a5d9a29b2ec P b56de26098334552bb9ae268d771c311: Bootstrap starting.
I20260923 20:08:54.411641 3335 tablet_bootstrap.cc:492] T 0df49309d341414aaf775a5d9a29b2ec P f1669496db8e44798ae7354b2d1f315b: No bootstrap required, opened a new log
I20260923 20:08:54.411864 3335 ts_tablet_manager.cc:1397] T 0df49309d341414aaf775a5d9a29b2ec P f1669496db8e44798ae7354b2d1f315b: Time spent bootstrapping tablet: real 0.013s user 0.001s sys 0.009s
I20260923 20:08:54.412940 3337 tablet_bootstrap.cc:654] T 0df49309d341414aaf775a5d9a29b2ec P b56de26098334552bb9ae268d771c311: Neither blocks nor log segments found. Creating new log.
I20260923 20:08:54.412510 3335 raft_consensus.cc:348] T 0df49309d341414aaf775a5d9a29b2ec P f1669496db8e44798ae7354b2d1f315b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1669496db8e44798ae7354b2d1f315b" member_type: VOTER last_known_addr { host: "127.1.170.129" port: 38261 } } peers { permanent_uuid: "b56de26098334552bb9ae268d771c311" member_type: VOTER last_known_addr { host: "127.1.170.130" port: 33471 } } peers { permanent_uuid: "42013bee84ed41b59532dc3b7fd90adb" member_type: VOTER last_known_addr { host: "127.1.170.131" port: 44771 } }
I20260923 20:08:54.413239 3335 raft_consensus.cc:374] T 0df49309d341414aaf775a5d9a29b2ec P f1669496db8e44798ae7354b2d1f315b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:08:54.413462 3335 raft_consensus.cc:729] T 0df49309d341414aaf775a5d9a29b2ec P f1669496db8e44798ae7354b2d1f315b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1669496db8e44798ae7354b2d1f315b, State: Initialized, Role: FOLLOWER
I20260923 20:08:54.413841 3335 consensus_queue.cc:260] T 0df49309d341414aaf775a5d9a29b2ec P f1669496db8e44798ae7354b2d1f315b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1669496db8e44798ae7354b2d1f315b" member_type: VOTER last_known_addr { host: "127.1.170.129" port: 38261 } } peers { permanent_uuid: "b56de26098334552bb9ae268d771c311" member_type: VOTER last_known_addr { host: "127.1.170.130" port: 33471 } } peers { permanent_uuid: "42013bee84ed41b59532dc3b7fd90adb" member_type: VOTER last_known_addr { host: "127.1.170.131" port: 44771 } }
I20260923 20:08:54.416034 3337 tablet_bootstrap.cc:492] T 0df49309d341414aaf775a5d9a29b2ec P b56de26098334552bb9ae268d771c311: No bootstrap required, opened a new log
I20260923 20:08:54.416034 3336 tablet_bootstrap.cc:492] T 0df49309d341414aaf775a5d9a29b2ec P 42013bee84ed41b59532dc3b7fd90adb: No bootstrap required, opened a new log
I20260923 20:08:54.416220 3337 ts_tablet_manager.cc:1397] T 0df49309d341414aaf775a5d9a29b2ec P b56de26098334552bb9ae268d771c311: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20260923 20:08:54.416225 3336 ts_tablet_manager.cc:1397] T 0df49309d341414aaf775a5d9a29b2ec P 42013bee84ed41b59532dc3b7fd90adb: Time spent bootstrapping tablet: real 0.014s user 0.000s sys 0.008s
I20260923 20:08:54.416893 3336 raft_consensus.cc:348] T 0df49309d341414aaf775a5d9a29b2ec P 42013bee84ed41b59532dc3b7fd90adb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1669496db8e44798ae7354b2d1f315b" member_type: VOTER last_known_addr { host: "127.1.170.129" port: 38261 } } peers { permanent_uuid: "b56de26098334552bb9ae268d771c311" member_type: VOTER last_known_addr { host: "127.1.170.130" port: 33471 } } peers { permanent_uuid: "42013bee84ed41b59532dc3b7fd90adb" member_type: VOTER last_known_addr { host: "127.1.170.131" port: 44771 } }
I20260923 20:08:54.417114 3336 raft_consensus.cc:374] T 0df49309d341414aaf775a5d9a29b2ec P 42013bee84ed41b59532dc3b7fd90adb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:08:54.416989 3337 raft_consensus.cc:348] T 0df49309d341414aaf775a5d9a29b2ec P b56de26098334552bb9ae268d771c311 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1669496db8e44798ae7354b2d1f315b" member_type: VOTER last_known_addr { host: "127.1.170.129" port: 38261 } } peers { permanent_uuid: "b56de26098334552bb9ae268d771c311" member_type: VOTER last_known_addr { host: "127.1.170.130" port: 33471 } } peers { permanent_uuid: "42013bee84ed41b59532dc3b7fd90adb" member_type: VOTER last_known_addr { host: "127.1.170.131" port: 44771 } }
I20260923 20:08:54.417193 3336 raft_consensus.cc:729] T 0df49309d341414aaf775a5d9a29b2ec P 42013bee84ed41b59532dc3b7fd90adb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 42013bee84ed41b59532dc3b7fd90adb, State: Initialized, Role: FOLLOWER
I20260923 20:08:54.417248 3337 raft_consensus.cc:374] T 0df49309d341414aaf775a5d9a29b2ec P b56de26098334552bb9ae268d771c311 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:08:54.417338 3337 raft_consensus.cc:729] T 0df49309d341414aaf775a5d9a29b2ec P b56de26098334552bb9ae268d771c311 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b56de26098334552bb9ae268d771c311, State: Initialized, Role: FOLLOWER
I20260923 20:08:54.414629 3357 tablet_replica.cc:439] T 0df49309d341414aaf775a5d9a29b2ec P f1669496db8e44798ae7354b2d1f315b: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1669496db8e44798ae7354b2d1f315b" member_type: VOTER last_known_addr { host: "127.1.170.129" port: 38261 } } peers { permanent_uuid: "b56de26098334552bb9ae268d771c311" member_type: VOTER last_known_addr { host: "127.1.170.130" port: 33471 } } peers { permanent_uuid: "42013bee84ed41b59532dc3b7fd90adb" member_type: VOTER last_known_addr { host: "127.1.170.131" port: 44771 } } }
I20260923 20:08:54.417629 3357 tablet_replica.cc:442] T 0df49309d341414aaf775a5d9a29b2ec P f1669496db8e44798ae7354b2d1f315b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260923 20:08:54.417577 3337 consensus_queue.cc:260] T 0df49309d341414aaf775a5d9a29b2ec P b56de26098334552bb9ae268d771c311 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1669496db8e44798ae7354b2d1f315b" member_type: VOTER last_known_addr { host: "127.1.170.129" port: 38261 } } peers { permanent_uuid: "b56de26098334552bb9ae268d771c311" member_type: VOTER last_known_addr { host: "127.1.170.130" port: 33471 } } peers { permanent_uuid: "42013bee84ed41b59532dc3b7fd90adb" member_type: VOTER last_known_addr { host: "127.1.170.131" port: 44771 } }
I20260923 20:08:54.418053 3335 ts_tablet_manager.cc:1428] T 0df49309d341414aaf775a5d9a29b2ec P f1669496db8e44798ae7354b2d1f315b: Time spent starting tablet: real 0.006s user 0.003s sys 0.000s
I20260923 20:08:54.418136 3337 ts_tablet_manager.cc:1428] T 0df49309d341414aaf775a5d9a29b2ec P b56de26098334552bb9ae268d771c311: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260923 20:08:54.418159 3341 tablet_replica.cc:439] T 0df49309d341414aaf775a5d9a29b2ec P b56de26098334552bb9ae268d771c311: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1669496db8e44798ae7354b2d1f315b" member_type: VOTER last_known_addr { host: "127.1.170.129" port: 38261 } } peers { permanent_uuid: "b56de26098334552bb9ae268d771c311" member_type: VOTER last_known_addr { host: "127.1.170.130" port: 33471 } } peers { permanent_uuid: "42013bee84ed41b59532dc3b7fd90adb" member_type: VOTER last_known_addr { host: "127.1.170.131" port: 44771 } } }
I20260923 20:08:54.418478 3341 tablet_replica.cc:442] T 0df49309d341414aaf775a5d9a29b2ec P b56de26098334552bb9ae268d771c311: This TxnStatusTablet replica's current role is: FOLLOWER
I20260923 20:08:54.419135 3336 consensus_queue.cc:260] T 0df49309d341414aaf775a5d9a29b2ec P 42013bee84ed41b59532dc3b7fd90adb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1669496db8e44798ae7354b2d1f315b" member_type: VOTER last_known_addr { host: "127.1.170.129" port: 38261 } } peers { permanent_uuid: "b56de26098334552bb9ae268d771c311" member_type: VOTER last_known_addr { host: "127.1.170.130" port: 33471 } } peers { permanent_uuid: "42013bee84ed41b59532dc3b7fd90adb" member_type: VOTER last_known_addr { host: "127.1.170.131" port: 44771 } }
I20260923 20:08:54.419828 3336 ts_tablet_manager.cc:1428] T 0df49309d341414aaf775a5d9a29b2ec P 42013bee84ed41b59532dc3b7fd90adb: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20260923 20:08:54.419704 3342 tablet_replica.cc:439] T 0df49309d341414aaf775a5d9a29b2ec P 42013bee84ed41b59532dc3b7fd90adb: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1669496db8e44798ae7354b2d1f315b" member_type: VOTER last_known_addr { host: "127.1.170.129" port: 38261 } } peers { permanent_uuid: "b56de26098334552bb9ae268d771c311" member_type: VOTER last_known_addr { host: "127.1.170.130" port: 33471 } } peers { permanent_uuid: "42013bee84ed41b59532dc3b7fd90adb" member_type: VOTER last_known_addr { host: "127.1.170.131" port: 44771 } } }
I20260923 20:08:54.420051 3342 tablet_replica.cc:442] T 0df49309d341414aaf775a5d9a29b2ec P 42013bee84ed41b59532dc3b7fd90adb: This TxnStatusTablet replica's current role is: FOLLOWER
I20260923 20:08:54.634039 3357 raft_consensus.cc:482] T 0df49309d341414aaf775a5d9a29b2ec P f1669496db8e44798ae7354b2d1f315b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 20:08:54.634232 3357 raft_consensus.cc:504] T 0df49309d341414aaf775a5d9a29b2ec P f1669496db8e44798ae7354b2d1f315b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1669496db8e44798ae7354b2d1f315b" member_type: VOTER last_known_addr { host: "127.1.170.129" port: 38261 } } peers { permanent_uuid: "b56de26098334552bb9ae268d771c311" member_type: VOTER last_known_addr { host: "127.1.170.130" port: 33471 } } peers { permanent_uuid: "42013bee84ed41b59532dc3b7fd90adb" member_type: VOTER last_known_addr { host: "127.1.170.131" port: 44771 } }
I20260923 20:08:54.634929 3357 leader_election.cc:288] T 0df49309d341414aaf775a5d9a29b2ec P f1669496db8e44798ae7354b2d1f315b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b56de26098334552bb9ae268d771c311 (127.1.170.130:33471), 42013bee84ed41b59532dc3b7fd90adb (127.1.170.131:44771)
I20260923 20:08:54.635802 3127 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0df49309d341414aaf775a5d9a29b2ec" candidate_uuid: "f1669496db8e44798ae7354b2d1f315b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b56de26098334552bb9ae268d771c311" is_pre_election: true
I20260923 20:08:54.635929 3270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0df49309d341414aaf775a5d9a29b2ec" candidate_uuid: "f1669496db8e44798ae7354b2d1f315b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "42013bee84ed41b59532dc3b7fd90adb" is_pre_election: true
I20260923 20:08:54.636068 3127 raft_consensus.cc:2454] T 0df49309d341414aaf775a5d9a29b2ec P b56de26098334552bb9ae268d771c311 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f1669496db8e44798ae7354b2d1f315b in term 0.
I20260923 20:08:54.636438 3270 raft_consensus.cc:2454] T 0df49309d341414aaf775a5d9a29b2ec P 42013bee84ed41b59532dc3b7fd90adb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f1669496db8e44798ae7354b2d1f315b in term 0.
I20260923 20:08:54.636871 2921 leader_election.cc:302] T 0df49309d341414aaf775a5d9a29b2ec P f1669496db8e44798ae7354b2d1f315b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b56de26098334552bb9ae268d771c311, f1669496db8e44798ae7354b2d1f315b; no voters:
I20260923 20:08:54.637199 3357 raft_consensus.cc:2781] T 0df49309d341414aaf775a5d9a29b2ec P f1669496db8e44798ae7354b2d1f315b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 20:08:54.637326 3357 raft_consensus.cc:482] T 0df49309d341414aaf775a5d9a29b2ec P f1669496db8e44798ae7354b2d1f315b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 20:08:54.637432 3357 raft_consensus.cc:3037] T 0df49309d341414aaf775a5d9a29b2ec P f1669496db8e44798ae7354b2d1f315b [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:08:54.639444 3357 raft_consensus.cc:504] T 0df49309d341414aaf775a5d9a29b2ec P f1669496db8e44798ae7354b2d1f315b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1669496db8e44798ae7354b2d1f315b" member_type: VOTER last_known_addr { host: "127.1.170.129" port: 38261 } } peers { permanent_uuid: "b56de26098334552bb9ae268d771c311" member_type: VOTER last_known_addr { host: "127.1.170.130" port: 33471 } } peers { permanent_uuid: "42013bee84ed41b59532dc3b7fd90adb" member_type: VOTER last_known_addr { host: "127.1.170.131" port: 44771 } }
I20260923 20:08:54.640035 3357 leader_election.cc:288] T 0df49309d341414aaf775a5d9a29b2ec P f1669496db8e44798ae7354b2d1f315b [CANDIDATE]: Term 1 election: Requested vote from peers b56de26098334552bb9ae268d771c311 (127.1.170.130:33471), 42013bee84ed41b59532dc3b7fd90adb (127.1.170.131:44771)
I20260923 20:08:54.640599 3127 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0df49309d341414aaf775a5d9a29b2ec" candidate_uuid: "f1669496db8e44798ae7354b2d1f315b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b56de26098334552bb9ae268d771c311"
I20260923 20:08:54.640822 3127 raft_consensus.cc:3037] T 0df49309d341414aaf775a5d9a29b2ec P b56de26098334552bb9ae268d771c311 [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:08:54.640898 3270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0df49309d341414aaf775a5d9a29b2ec" candidate_uuid: "f1669496db8e44798ae7354b2d1f315b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "42013bee84ed41b59532dc3b7fd90adb"
I20260923 20:08:54.641369 3270 raft_consensus.cc:3037] T 0df49309d341414aaf775a5d9a29b2ec P 42013bee84ed41b59532dc3b7fd90adb [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:08:54.642922 3127 raft_consensus.cc:2454] T 0df49309d341414aaf775a5d9a29b2ec P b56de26098334552bb9ae268d771c311 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f1669496db8e44798ae7354b2d1f315b in term 1.
I20260923 20:08:54.643390 2921 leader_election.cc:302] T 0df49309d341414aaf775a5d9a29b2ec P f1669496db8e44798ae7354b2d1f315b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b56de26098334552bb9ae268d771c311, f1669496db8e44798ae7354b2d1f315b; no voters:
I20260923 20:08:54.643703 3357 raft_consensus.cc:2781] T 0df49309d341414aaf775a5d9a29b2ec P f1669496db8e44798ae7354b2d1f315b [term 1 FOLLOWER]: Leader election won for term 1
I20260923 20:08:54.644155 3357 raft_consensus.cc:686] T 0df49309d341414aaf775a5d9a29b2ec P f1669496db8e44798ae7354b2d1f315b [term 1 LEADER]: Becoming Leader. State: Replica: f1669496db8e44798ae7354b2d1f315b, State: Running, Role: LEADER
I20260923 20:08:54.644452 3357 consensus_queue.cc:237] T 0df49309d341414aaf775a5d9a29b2ec P f1669496db8e44798ae7354b2d1f315b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1669496db8e44798ae7354b2d1f315b" member_type: VOTER last_known_addr { host: "127.1.170.129" port: 38261 } } peers { permanent_uuid: "b56de26098334552bb9ae268d771c311" member_type: VOTER last_known_addr { host: "127.1.170.130" port: 33471 } } peers { permanent_uuid: "42013bee84ed41b59532dc3b7fd90adb" member_type: VOTER last_known_addr { host: "127.1.170.131" port: 44771 } }
I20260923 20:08:54.644706 3270 raft_consensus.cc:2454] T 0df49309d341414aaf775a5d9a29b2ec P 42013bee84ed41b59532dc3b7fd90adb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f1669496db8e44798ae7354b2d1f315b in term 1.
I20260923 20:08:54.646435 3380 tablet_replica.cc:439] T 0df49309d341414aaf775a5d9a29b2ec P f1669496db8e44798ae7354b2d1f315b: TxnStatusTablet state changed. Reason: New leader f1669496db8e44798ae7354b2d1f315b. Latest consensus state: current_term: 1 leader_uuid: "f1669496db8e44798ae7354b2d1f315b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1669496db8e44798ae7354b2d1f315b" member_type: VOTER last_known_addr { host: "127.1.170.129" port: 38261 } } peers { permanent_uuid: "b56de26098334552bb9ae268d771c311" member_type: VOTER last_known_addr { host: "127.1.170.130" port: 33471 } } peers { permanent_uuid: "42013bee84ed41b59532dc3b7fd90adb" member_type: VOTER last_known_addr { host: "127.1.170.131" port: 44771 } } }
I20260923 20:08:54.646931 3380 tablet_replica.cc:442] T 0df49309d341414aaf775a5d9a29b2ec P f1669496db8e44798ae7354b2d1f315b: This TxnStatusTablet replica's current role is: LEADER
I20260923 20:08:54.647598 3382 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260923 20:08:54.647930 3382 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260923 20:08:54.655161 2828 catalog_manager.cc:5495] T 0df49309d341414aaf775a5d9a29b2ec P f1669496db8e44798ae7354b2d1f315b reported cstate change: term changed from 0 to 1, leader changed from <none> to f1669496db8e44798ae7354b2d1f315b (127.1.170.129). New cstate: current_term: 1 leader_uuid: "f1669496db8e44798ae7354b2d1f315b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1669496db8e44798ae7354b2d1f315b" member_type: VOTER last_known_addr { host: "127.1.170.129" port: 38261 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b56de26098334552bb9ae268d771c311" member_type: VOTER last_known_addr { host: "127.1.170.130" port: 33471 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "42013bee84ed41b59532dc3b7fd90adb" member_type: VOTER last_known_addr { host: "127.1.170.131" port: 44771 } health_report { overall_health: UNKNOWN } } }
I20260923 20:08:54.658288 3341 consensus_queue.cc:1035] T cdabeae2e5284ae9a299e9ad611006b6 P b56de26098334552bb9ae268d771c311 [LEADER]: Connected to new peer: Peer: permanent_uuid: "42013bee84ed41b59532dc3b7fd90adb" member_type: VOTER last_known_addr { host: "127.1.170.131" port: 44771 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 20:08:54.669953 3341 consensus_queue.cc:1035] T cdabeae2e5284ae9a299e9ad611006b6 P b56de26098334552bb9ae268d771c311 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1669496db8e44798ae7354b2d1f315b" member_type: VOTER last_known_addr { host: "127.1.170.129" port: 38261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 20:08:54.731684 3332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:08:54.724440 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52288 (local address 127.1.170.129:38261)
0923 20:08:54.724656 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0923 20:08:54.724662 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:08:54.724806 (+ 144us) server_negotiation.cc:407] Connection header received
0923 20:08:54.725154 (+ 348us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:08:54.725159 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:08:54.725262 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:08:54.725371 (+ 109us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 20:08:54.725845 (+ 474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:08:54.726743 (+ 898us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:08:54.729127 (+ 2384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:08:54.729652 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:08:54.729798 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:08:54.730417 (+ 619us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 20:08:54.730531 (+ 114us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 20:08:54.730609 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0923 20:08:54.731253 (+ 644us) server_negotiation.cc:299] Negotiation successful
0923 20:08:54.731363 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260923 20:08:54.731968 3376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:08:54.724417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.129:38261 (local address 127.0.0.1:52288)
0923 20:08:54.724690 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0923 20:08:54.724712 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0923 20:08:54.724936 (+ 224us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:08:54.725425 (+ 489us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:08:54.725431 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:08:54.725446 (+ 15us) client_negotiation.cc:189] Negotiated authn=TOKEN
0923 20:08:54.725662 (+ 216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:08:54.725674 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:08:54.726924 (+ 1250us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:08:54.726930 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:08:54.728946 (+ 2016us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:08:54.728963 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:08:54.729827 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:08:54.729834 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:08:54.730088 (+ 254us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:08:54.730174 (+ 86us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0923 20:08:54.730836 (+ 662us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0923 20:08:54.730887 (+ 51us) client_negotiation.cc:769] Sending connection context
0923 20:08:54.731096 (+ 209us) client_negotiation.cc:240] Negotiation successful
0923 20:08:54.731548 (+ 452us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":60}
I20260923 20:08:54.737289 3270 raft_consensus.cc:1261] T 0df49309d341414aaf775a5d9a29b2ec P 42013bee84ed41b59532dc3b7fd90adb [term 1 FOLLOWER]: Refusing update from remote peer f1669496db8e44798ae7354b2d1f315b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 20:08:54.737783 3127 raft_consensus.cc:1261] T 0df49309d341414aaf775a5d9a29b2ec P b56de26098334552bb9ae268d771c311 [term 1 FOLLOWER]: Refusing update from remote peer f1669496db8e44798ae7354b2d1f315b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 20:08:54.738004 3357 consensus_queue.cc:1035] T 0df49309d341414aaf775a5d9a29b2ec P f1669496db8e44798ae7354b2d1f315b [LEADER]: Connected to new peer: Peer: permanent_uuid: "42013bee84ed41b59532dc3b7fd90adb" member_type: VOTER last_known_addr { host: "127.1.170.131" port: 44771 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 20:08:54.738422 3380 consensus_queue.cc:1035] T 0df49309d341414aaf775a5d9a29b2ec P f1669496db8e44798ae7354b2d1f315b [LEADER]: Connected to new peer: Peer: permanent_uuid: "b56de26098334552bb9ae268d771c311" member_type: VOTER last_known_addr { host: "127.1.170.130" port: 33471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 20:08:54.742373 3342 tablet_replica.cc:439] T 0df49309d341414aaf775a5d9a29b2ec P 42013bee84ed41b59532dc3b7fd90adb: TxnStatusTablet state changed. Reason: New leader f1669496db8e44798ae7354b2d1f315b. Latest consensus state: current_term: 1 leader_uuid: "f1669496db8e44798ae7354b2d1f315b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1669496db8e44798ae7354b2d1f315b" member_type: VOTER last_known_addr { host: "127.1.170.129" port: 38261 } } peers { permanent_uuid: "b56de26098334552bb9ae268d771c311" member_type: VOTER last_known_addr { host: "127.1.170.130" port: 33471 } } peers { permanent_uuid: "42013bee84ed41b59532dc3b7fd90adb" member_type: VOTER last_known_addr { host: "127.1.170.131" port: 44771 } } }
I20260923 20:08:54.742842 3342 tablet_replica.cc:442] T 0df49309d341414aaf775a5d9a29b2ec P 42013bee84ed41b59532dc3b7fd90adb: This TxnStatusTablet replica's current role is: FOLLOWER
I20260923 20:08:54.743613 3341 tablet_replica.cc:439] T 0df49309d341414aaf775a5d9a29b2ec P b56de26098334552bb9ae268d771c311: TxnStatusTablet state changed. Reason: New leader f1669496db8e44798ae7354b2d1f315b. Latest consensus state: current_term: 1 leader_uuid: "f1669496db8e44798ae7354b2d1f315b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1669496db8e44798ae7354b2d1f315b" member_type: VOTER last_known_addr { host: "127.1.170.129" port: 38261 } } peers { permanent_uuid: "b56de26098334552bb9ae268d771c311" member_type: VOTER last_known_addr { host: "127.1.170.130" port: 33471 } } peers { permanent_uuid: "42013bee84ed41b59532dc3b7fd90adb" member_type: VOTER last_known_addr { host: "127.1.170.131" port: 44771 } } }
I20260923 20:08:54.743937 3341 tablet_replica.cc:442] T 0df49309d341414aaf775a5d9a29b2ec P b56de26098334552bb9ae268d771c311: This TxnStatusTablet replica's current role is: FOLLOWER
I20260923 20:08:54.744998 3388 mvcc.cc:205] Tried to move back new op lower bound from 7332635175876345856 to 7332635175506296832. Current Snapshot: MvccSnapshot[applied={T|T < 7332635175876345856}]
I20260923 20:08:54.747578 3341 tablet_replica.cc:439] T 0df49309d341414aaf775a5d9a29b2ec P b56de26098334552bb9ae268d771c311: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f1669496db8e44798ae7354b2d1f315b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1669496db8e44798ae7354b2d1f315b" member_type: VOTER last_known_addr { host: "127.1.170.129" port: 38261 } } peers { permanent_uuid: "b56de26098334552bb9ae268d771c311" member_type: VOTER last_known_addr { host: "127.1.170.130" port: 33471 } } peers { permanent_uuid: "42013bee84ed41b59532dc3b7fd90adb" member_type: VOTER last_known_addr { host: "127.1.170.131" port: 44771 } } }
I20260923 20:08:54.747865 3341 tablet_replica.cc:442] T 0df49309d341414aaf775a5d9a29b2ec P b56de26098334552bb9ae268d771c311: This TxnStatusTablet replica's current role is: FOLLOWER
I20260923 20:08:54.750658 3380 tablet_replica.cc:439] T 0df49309d341414aaf775a5d9a29b2ec P f1669496db8e44798ae7354b2d1f315b: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f1669496db8e44798ae7354b2d1f315b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1669496db8e44798ae7354b2d1f315b" member_type: VOTER last_known_addr { host: "127.1.170.129" port: 38261 } } peers { permanent_uuid: "b56de26098334552bb9ae268d771c311" member_type: VOTER last_known_addr { host: "127.1.170.130" port: 33471 } } peers { permanent_uuid: "42013bee84ed41b59532dc3b7fd90adb" member_type: VOTER last_known_addr { host: "127.1.170.131" port: 44771 } } }
I20260923 20:08:54.754457 3380 tablet_replica.cc:442] T 0df49309d341414aaf775a5d9a29b2ec P f1669496db8e44798ae7354b2d1f315b: This TxnStatusTablet replica's current role is: LEADER
I20260923 20:08:54.756011 3357 tablet_replica.cc:439] T 0df49309d341414aaf775a5d9a29b2ec P f1669496db8e44798ae7354b2d1f315b: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f1669496db8e44798ae7354b2d1f315b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1669496db8e44798ae7354b2d1f315b" member_type: VOTER last_known_addr { host: "127.1.170.129" port: 38261 } } peers { permanent_uuid: "b56de26098334552bb9ae268d771c311" member_type: VOTER last_known_addr { host: "127.1.170.130" port: 33471 } } peers { permanent_uuid: "42013bee84ed41b59532dc3b7fd90adb" member_type: VOTER last_known_addr { host: "127.1.170.131" port: 44771 } } }
I20260923 20:08:54.756456 3357 tablet_replica.cc:442] T 0df49309d341414aaf775a5d9a29b2ec P f1669496db8e44798ae7354b2d1f315b: This TxnStatusTablet replica's current role is: LEADER
I20260923 20:08:54.759136 3386 mvcc.cc:205] Tried to move back new op lower bound from 7332635175876345856 to 7332635175506296832. Current Snapshot: MvccSnapshot[applied={T|T < 7332635175876345856}]
I20260923 20:08:54.760398 3342 tablet_replica.cc:439] T 0df49309d341414aaf775a5d9a29b2ec P 42013bee84ed41b59532dc3b7fd90adb: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f1669496db8e44798ae7354b2d1f315b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1669496db8e44798ae7354b2d1f315b" member_type: VOTER last_known_addr { host: "127.1.170.129" port: 38261 } } peers { permanent_uuid: "b56de26098334552bb9ae268d771c311" member_type: VOTER last_known_addr { host: "127.1.170.130" port: 33471 } } peers { permanent_uuid: "42013bee84ed41b59532dc3b7fd90adb" member_type: VOTER last_known_addr { host: "127.1.170.131" port: 44771 } } }
I20260923 20:08:54.760656 3342 tablet_replica.cc:442] T 0df49309d341414aaf775a5d9a29b2ec P 42013bee84ed41b59532dc3b7fd90adb: This TxnStatusTablet replica's current role is: FOLLOWER
I20260923 20:08:54.773975 3397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:08:54.765844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51178 (local address 127.1.170.130:33471)
0923 20:08:54.766348 (+ 504us) server_negotiation.cc:206] Beginning negotiation
0923 20:08:54.766408 (+ 60us) server_negotiation.cc:399] Waiting for connection header
0923 20:08:54.766862 (+ 454us) server_negotiation.cc:407] Connection header received
0923 20:08:54.767069 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:08:54.767075 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:08:54.767196 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:08:54.767383 (+ 187us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 20:08:54.768245 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:08:54.769352 (+ 1107us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:08:54.771032 (+ 1680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:08:54.771717 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:08:54.771859 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:08:54.772285 (+ 426us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 20:08:54.773081 (+ 796us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 20:08:54.773252 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0923 20:08:54.773529 (+ 277us) server_negotiation.cc:299] Negotiation successful
0923 20:08:54.773627 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":381,"thread_start_us":250,"threads_started":1}
I20260923 20:08:54.795967 3332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:08:54.787278 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.170.130:54393 (local address 127.1.170.129:38261)
0923 20:08:54.787569 (+ 291us) server_negotiation.cc:206] Beginning negotiation
0923 20:08:54.787578 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 20:08:54.788339 (+ 761us) server_negotiation.cc:407] Connection header received
0923 20:08:54.788648 (+ 309us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:08:54.788654 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:08:54.788743 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:08:54.788871 (+ 128us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 20:08:54.790162 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:08:54.791366 (+ 1204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:08:54.793351 (+ 1985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:08:54.794165 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:08:54.794322 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:08:54.794883 (+ 561us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 20:08:54.795050 (+ 167us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 20:08:54.795147 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0923 20:08:54.795493 (+ 346us) server_negotiation.cc:299] Negotiation successful
0923 20:08:54.795587 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260923 20:08:54.795998 3401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:08:54.786833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.129:38261 (local address 127.1.170.130:54393)
0923 20:08:54.787995 (+ 1162us) negotiation.cc:107] Waiting for socket to connect
0923 20:08:54.788016 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0923 20:08:54.788471 (+ 455us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:08:54.788889 (+ 418us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:08:54.788893 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:08:54.788906 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0923 20:08:54.789605 (+ 699us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:08:54.789614 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:08:54.791584 (+ 1970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:08:54.791589 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:08:54.793066 (+ 1477us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:08:54.793109 (+ 43us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:08:54.794388 (+ 1279us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:08:54.794394 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:08:54.794546 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:08:54.794635 (+ 89us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0923 20:08:54.795296 (+ 661us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0923 20:08:54.795314 (+ 18us) client_negotiation.cc:769] Sending connection context
0923 20:08:54.795440 (+ 126us) client_negotiation.cc:240] Negotiation successful
0923 20:08:54.795621 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":971,"thread_start_us":784,"threads_started":1}
I20260923 20:08:54.845801 3397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:08:54.838116 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.170.129:48795 (local address 127.1.170.130:33471)
0923 20:08:54.838486 (+ 370us) server_negotiation.cc:206] Beginning negotiation
0923 20:08:54.838492 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:08:54.838513 (+ 21us) server_negotiation.cc:407] Connection header received
0923 20:08:54.838735 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:08:54.838742 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:08:54.838835 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:08:54.838931 (+ 96us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 20:08:54.839448 (+ 517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:08:54.840585 (+ 1137us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:08:54.843220 (+ 2635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:08:54.843702 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:08:54.843814 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:08:54.844415 (+ 601us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 20:08:54.844608 (+ 193us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 20:08:54.844707 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0923 20:08:54.844952 (+ 245us) server_negotiation.cc:299] Negotiation successful
0923 20:08:54.845035 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":248}
I20260923 20:08:54.846127 3405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:08:54.837654 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.130:33471 (local address 127.1.170.129:48795)
0923 20:08:54.838409 (+ 755us) negotiation.cc:107] Waiting for socket to connect
0923 20:08:54.838441 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0923 20:08:54.838573 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:08:54.838950 (+ 377us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:08:54.838954 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:08:54.838966 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0923 20:08:54.839291 (+ 325us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:08:54.839299 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:08:54.840947 (+ 1648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:08:54.840953 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:08:54.843044 (+ 2091us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:08:54.843064 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:08:54.843899 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:08:54.843906 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:08:54.844082 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:08:54.844171 (+ 89us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0923 20:08:54.844708 (+ 537us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0923 20:08:54.844723 (+ 15us) client_negotiation.cc:769] Sending connection context
0923 20:08:54.844940 (+ 217us) client_negotiation.cc:240] Negotiation successful
0923 20:08:54.845091 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":491,"thread_start_us":496,"threads_started":1}
I20260923 20:08:54.896557 2828 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42432:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 20:08:54.896818 2828 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42432:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 20:08:54.900708 2828 catalog_manager.cc:5782] T 00000000000000000000000000000000 P be44f549a3ba42f48a99b54e16576e8a: Sending DeleteTablet for 3 replicas of tablet cdabeae2e5284ae9a299e9ad611006b6
I20260923 20:08:54.902168 3245 tablet_service.cc:1514] Processing DeleteTablet for tablet cdabeae2e5284ae9a299e9ad611006b6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 20:08:54 UTC) from {username='kudu', principal='kudu/127.1.170.190@KRBTEST.COM'} at 127.0.0.1:35654
I20260923 20:08:54.902892 2967 tablet_service.cc:1514] Processing DeleteTablet for tablet cdabeae2e5284ae9a299e9ad611006b6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 20:08:54 UTC) from {username='kudu', principal='kudu/127.1.170.190@KRBTEST.COM'} at 127.0.0.1:52274
I20260923 20:08:54.907006 3107 tablet_service.cc:1514] Processing DeleteTablet for tablet cdabeae2e5284ae9a299e9ad611006b6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 20:08:54 UTC) from {username='kudu', principal='kudu/127.1.170.190@KRBTEST.COM'} at 127.0.0.1:51168
I20260923 20:08:54.916894 3408 tablet_replica.cc:330] T cdabeae2e5284ae9a299e9ad611006b6 P 42013bee84ed41b59532dc3b7fd90adb: stopping tablet replica
I20260923 20:08:54.917986 3409 tablet_replica.cc:330] T cdabeae2e5284ae9a299e9ad611006b6 P f1669496db8e44798ae7354b2d1f315b: stopping tablet replica
I20260923 20:08:54.918557 3411 tablet_replica.cc:330] T cdabeae2e5284ae9a299e9ad611006b6 P b56de26098334552bb9ae268d771c311: stopping tablet replica
I20260923 20:08:54.919356 3408 raft_consensus.cc:2229] T cdabeae2e5284ae9a299e9ad611006b6 P 42013bee84ed41b59532dc3b7fd90adb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 20:08:54.922417 3411 raft_consensus.cc:2229] T cdabeae2e5284ae9a299e9ad611006b6 P b56de26098334552bb9ae268d771c311 [term 1 LEADER]: Raft consensus shutting down.
I20260923 20:08:54.923027 3408 raft_consensus.cc:2258] T cdabeae2e5284ae9a299e9ad611006b6 P 42013bee84ed41b59532dc3b7fd90adb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 20:08:54.923436 3409 raft_consensus.cc:2229] T cdabeae2e5284ae9a299e9ad611006b6 P f1669496db8e44798ae7354b2d1f315b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 20:08:54.925127 3409 raft_consensus.cc:2258] T cdabeae2e5284ae9a299e9ad611006b6 P f1669496db8e44798ae7354b2d1f315b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 20:08:54.927034 3409 ts_tablet_manager.cc:1905] T cdabeae2e5284ae9a299e9ad611006b6 P f1669496db8e44798ae7354b2d1f315b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 20:08:54.927224 3408 ts_tablet_manager.cc:1905] T cdabeae2e5284ae9a299e9ad611006b6 P 42013bee84ed41b59532dc3b7fd90adb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 20:08:54.932404 3411 raft_consensus.cc:2258] T cdabeae2e5284ae9a299e9ad611006b6 P b56de26098334552bb9ae268d771c311 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 20:08:54.933655 3411 ts_tablet_manager.cc:1905] T cdabeae2e5284ae9a299e9ad611006b6 P b56de26098334552bb9ae268d771c311: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 20:08:54.938971 3409 ts_tablet_manager.cc:1918] T cdabeae2e5284ae9a299e9ad611006b6 P f1669496db8e44798ae7354b2d1f315b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260923 20:08:54.939291 3409 log.cc:1196] T cdabeae2e5284ae9a299e9ad611006b6 P f1669496db8e44798ae7354b2d1f315b: Deleting WAL directory at /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790194121577027-1706-0/minicluster-data/ts-0/wal/wals/cdabeae2e5284ae9a299e9ad611006b6
I20260923 20:08:54.940084 3409 ts_tablet_manager.cc:1939] T cdabeae2e5284ae9a299e9ad611006b6 P f1669496db8e44798ae7354b2d1f315b: Deleting consensus metadata
I20260923 20:08:54.941769 2815 catalog_manager.cc:4841] TS f1669496db8e44798ae7354b2d1f315b (127.1.170.129:38261): tablet cdabeae2e5284ae9a299e9ad611006b6 (table test-table [id=fba67baa5f5045058d8917ff2c0d69f3]) successfully deleted
I20260923 20:08:54.962596 3408 ts_tablet_manager.cc:1918] T cdabeae2e5284ae9a299e9ad611006b6 P 42013bee84ed41b59532dc3b7fd90adb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260923 20:08:54.963018 3408 log.cc:1196] T cdabeae2e5284ae9a299e9ad611006b6 P 42013bee84ed41b59532dc3b7fd90adb: Deleting WAL directory at /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790194121577027-1706-0/minicluster-data/ts-2/wal/wals/cdabeae2e5284ae9a299e9ad611006b6
I20260923 20:08:54.963693 3408 ts_tablet_manager.cc:1939] T cdabeae2e5284ae9a299e9ad611006b6 P 42013bee84ed41b59532dc3b7fd90adb: Deleting consensus metadata
I20260923 20:08:54.966097 2818 catalog_manager.cc:4841] TS 42013bee84ed41b59532dc3b7fd90adb (127.1.170.131:44771): tablet cdabeae2e5284ae9a299e9ad611006b6 (table test-table [id=fba67baa5f5045058d8917ff2c0d69f3]) successfully deleted
I20260923 20:08:54.966367 3411 ts_tablet_manager.cc:1918] T cdabeae2e5284ae9a299e9ad611006b6 P b56de26098334552bb9ae268d771c311: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260923 20:08:54.966650 3411 log.cc:1196] T cdabeae2e5284ae9a299e9ad611006b6 P b56de26098334552bb9ae268d771c311: Deleting WAL directory at /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790194121577027-1706-0/minicluster-data/ts-1/wal/wals/cdabeae2e5284ae9a299e9ad611006b6
I20260923 20:08:54.967168 3411 ts_tablet_manager.cc:1939] T cdabeae2e5284ae9a299e9ad611006b6 P b56de26098334552bb9ae268d771c311: Deleting consensus metadata
I20260923 20:08:54.968375 2818 catalog_manager.cc:4841] TS b56de26098334552bb9ae268d771c311 (127.1.170.130:33471): tablet cdabeae2e5284ae9a299e9ad611006b6 (table test-table [id=fba67baa5f5045058d8917ff2c0d69f3]) successfully deleted
Sep 23 20:08:54 dist-test-slave-th4g krb5kdc[2786](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194133, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.1.170.129@KRBTEST.COM
I20260923 20:08:54.982689 3332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:08:54.925532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52290 (local address 127.1.170.129:38261)
0923 20:08:54.925742 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0923 20:08:54.925754 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0923 20:08:54.966013 (+ 40259us) server_negotiation.cc:407] Connection header received
0923 20:08:54.966507 (+ 494us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:08:54.966519 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:08:54.966610 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:08:54.966740 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:08:54.968919 (+ 2179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:08:54.969702 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:08:54.971035 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:08:54.971482 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:08:54.971622 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:08:54.975706 (+ 4084us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:08:54.975721 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:08:54.975737 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:08:54.975776 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:08:54.979548 (+ 3772us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:08:54.980376 (+ 828us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:08:54.980383 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:08:54.980386 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:08:54.980522 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:08:54.980943 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:08:54.980947 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:08:54.980949 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:08:54.981340 (+ 391us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:08:54.981643 (+ 303us) server_negotiation.cc:1035] Waiting for connection context
0923 20:08:54.981987 (+ 344us) server_negotiation.cc:299] Negotiation successful
0923 20:08:54.982253 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
W20260923 20:08:54.983716 3033 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52290
I20260923 20:08:55.009622 3414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:08:54.987107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42440 (local address 127.1.170.190:39075)
0923 20:08:54.988046 (+ 939us) server_negotiation.cc:206] Beginning negotiation
0923 20:08:54.988053 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:08:54.988543 (+ 490us) server_negotiation.cc:407] Connection header received
0923 20:08:54.988739 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:08:54.988745 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:08:54.988865 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:08:54.988971 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:08:54.991685 (+ 2714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:08:54.993400 (+ 1715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:08:54.995948 (+ 2548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:08:54.996854 (+ 906us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:08:54.997259 (+ 405us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:08:55.002053 (+ 4794us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:08:55.002059 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:08:55.002062 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:08:55.002103 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:08:55.005608 (+ 3505us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:08:55.006659 (+ 1051us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:08:55.006665 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:08:55.006668 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:08:55.006873 (+ 205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:08:55.007365 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:08:55.007369 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:08:55.007372 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:08:55.007750 (+ 378us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:08:55.008074 (+ 324us) server_negotiation.cc:1035] Waiting for connection context
0923 20:08:55.008826 (+ 752us) server_negotiation.cc:299] Negotiation successful
0923 20:08:55.009162 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":818,"thread_start_us":660,"threads_started":1}
W20260923 20:08:55.011265 2828 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42440
I20260923 20:08:55.014127 1706 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskqUuku6/build/asan/bin/kudu pid 2902
I20260923 20:08:55.033416 3332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:08:55.015893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52300 (local address 127.1.170.129:38261)
0923 20:08:55.016100 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0923 20:08:55.016106 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:08:55.016583 (+ 477us) server_negotiation.cc:407] Connection header received
0923 20:08:55.016816 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:08:55.016821 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:08:55.016914 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:08:55.017042 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:08:55.018294 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:08:55.019577 (+ 1283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:08:55.021499 (+ 1922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:08:55.022290 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:08:55.022422 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:08:55.025268 (+ 2846us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:08:55.025274 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:08:55.025277 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:08:55.025358 (+ 81us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:08:55.029043 (+ 3685us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:08:55.030235 (+ 1192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:08:55.030242 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:08:55.030246 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:08:55.030613 (+ 367us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:08:55.031051 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:08:55.031056 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:08:55.031060 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:08:55.031650 (+ 590us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:08:55.031994 (+ 344us) server_negotiation.cc:1035] Waiting for connection context
0923 20:08:55.032580 (+ 586us) server_negotiation.cc:299] Negotiation successful
0923 20:08:55.032869 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
W20260923 20:08:55.034147 3033 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52300
W20260923 20:08:55.034749 1706 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskqUuku6/build/asan/bin/kudu pid 2902: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260923 20:08:55.034950 1706 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqUuku6/build/asan/bin/kudu with pid 2902
I20260923 20:08:55.069159 1706 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskqUuku6/build/asan/bin/kudu pid 3042
Sep 23 20:08:55 dist-test-slave-th4g krb5kdc[2786](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194133, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.1.170.130@KRBTEST.COM
I20260923 20:08:55.089154 3397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:08:55.070202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51188 (local address 127.1.170.130:33471)
0923 20:08:55.070385 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0923 20:08:55.070393 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 20:08:55.070410 (+ 17us) server_negotiation.cc:407] Connection header received
0923 20:08:55.070482 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:08:55.070486 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:08:55.070562 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:08:55.070683 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:08:55.072160 (+ 1477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:08:55.073325 (+ 1165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:08:55.075282 (+ 1957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:08:55.076001 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:08:55.076141 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:08:55.081398 (+ 5257us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:08:55.081415 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:08:55.081437 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:08:55.081488 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:08:55.085918 (+ 4430us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:08:55.086832 (+ 914us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:08:55.086847 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:08:55.086852 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:08:55.087021 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:08:55.087430 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:08:55.087435 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:08:55.087439 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:08:55.087928 (+ 489us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:08:55.088212 (+ 284us) server_negotiation.cc:1035] Waiting for connection context
0923 20:08:55.088532 (+ 320us) server_negotiation.cc:299] Negotiation successful
0923 20:08:55.088770 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
W20260923 20:08:55.090257 3172 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51188
W20260923 20:08:55.090951 1706 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskqUuku6/build/asan/bin/kudu pid 3042: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260923 20:08:55.091137 1706 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqUuku6/build/asan/bin/kudu with pid 3042
I20260923 20:08:55.109488 1706 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskqUuku6/build/asan/bin/kudu pid 3181
Sep 23 20:08:55 dist-test-slave-th4g krb5kdc[2786](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194133, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.1.170.131@KRBTEST.COM
I20260923 20:08:55.126636 3417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:08:55.110412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35666 (local address 127.1.170.131:44771)
0923 20:08:55.110979 (+ 567us) server_negotiation.cc:206] Beginning negotiation
0923 20:08:55.110984 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 20:08:55.111000 (+ 16us) server_negotiation.cc:407] Connection header received
0923 20:08:55.111052 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:08:55.111055 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:08:55.111111 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:08:55.111206 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:08:55.112383 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:08:55.113246 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:08:55.114782 (+ 1536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:08:55.115191 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:08:55.115306 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:08:55.120159 (+ 4853us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:08:55.120175 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:08:55.120193 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:08:55.120232 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:08:55.123666 (+ 3434us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:08:55.124449 (+ 783us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:08:55.124465 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:08:55.124468 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:08:55.124625 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:08:55.125023 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:08:55.125027 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:08:55.125029 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:08:55.125416 (+ 387us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:08:55.125659 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0923 20:08:55.126040 (+ 381us) server_negotiation.cc:299] Negotiation successful
0923 20:08:55.126211 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":464,"thread_start_us":308,"threads_started":1}
W20260923 20:08:55.127391 3311 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35666
W20260923 20:08:55.128063 1706 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskqUuku6/build/asan/bin/kudu pid 3181: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260923 20:08:55.128186 1706 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqUuku6/build/asan/bin/kudu with pid 3181
I20260923 20:08:55.146166 1706 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskqUuku6/build/asan/bin/kudu pid 2799
I20260923 20:08:55.147403 2871 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:08:55.267658 1706 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqUuku6/build/asan/bin/kudu with pid 2799
2026-09-23T20:08:55Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (5119 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790194121577027-1706-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 20:08:55 dist-test-slave-th4g krb5kdc[3421](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 20:08:55 dist-test-slave-th4g krb5kdc[3421](info): set up 2 sockets
Sep 23 20:08:55 dist-test-slave-th4g krb5kdc[3421](info): commencing operation
krb5kdc: starting...
W20260923 20:08:56.374279 1706 mini_kdc.cc:121] Time spent starting KDC: real 1.062s user 0.001s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790194121577027-1706-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790194121577027-1706-0/krb5kdc/test-user.keytab.
Sep 23 20:08:56 dist-test-slave-th4g krb5kdc[3421](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194136, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T20:08:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T20:08:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.170.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.170.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.170.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790194121577027-1706-0/krb5kdc/kudu_127.1.170.190.keytab.
Entry for principal kudu/127.1.170.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790194121577027-1706-0/krb5kdc/kudu_127.1.170.190.keytab.
I20260923 20:08:56.561682 1706 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqUuku6/build/asan/bin/kudu
/tmp/dist-test-taskqUuku6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790194121577027-1706-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790194121577027-1706-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790194121577027-1706-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790194121577027-1706-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.170.190:35033
--webserver_interface=127.1.170.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790194121577027-1706-0/krb5kdc/kudu_127.1.170.190.keytab
--principal=kudu/127.1.170.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.170.148:43717
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.170.190:35033
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790194121577027-1706-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790194121577027-1706-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790194121577027-1706-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:08:56.878970 3434 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:08:56.879354 3434 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:08:56.879520 3434 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:08:56.886885 3434 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 20:08:56.887043 3434 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260923 20:08:56.887140 3434 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:08:56.887251 3434 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:08:56.887346 3434 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 20:08:56.887434 3434 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 20:08:56.897205 3434 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.170.148:43717
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790194121577027-1706-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790194121577027-1706-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.170.190:35033
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790194121577027-1706-0/krb5kdc/kudu_127.1.170.190.keytab
--principal=kudu/127.1.170.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.170.190:35033
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790194121577027-1706-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.170.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790194121577027-1706-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:08:56.898559 3434 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:08:56.899916 3434 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:08:56.911542 3439 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:08:56.911573 3442 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:08:56.911618 3440 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:08:56.912173 3434 server_base.cc:1034] running on GCE node
I20260923 20:08:56.912982 3434 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:08:56.914469 3434 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:08:56.915764 3434 hybrid_clock.cc:648] HybridClock initialized: now 1790194136915684 us; error 51 us; skew 500 ppm
I20260923 20:08:56.916129 3434 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:08:56 dist-test-slave-th4g krb5kdc[3421](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194136, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.170.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:08:56.919533 3434 init.cc:377] Logged in from keytab as kudu/127.1.170.190@KRBTEST.COM (short username kudu)
I20260923 20:08:56.921126 3434 webserver.cc:460] Webserver started at http://127.1.170.190:41105/ using document root <none> and password file <none>
I20260923 20:08:56.921849 3434 fs_manager.cc:359] Metadata directory not provided
I20260923 20:08:56.922035 3434 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:08:56.922361 3434 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:08:56.925516 3434 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790194121577027-1706-0/minicluster-data/master-0/data/instance:
uuid: "4d74cac2e4f944208b16833320585dd4"
format_stamp: "Formatted at 2026-09-23 20:08:56 on dist-test-slave-th4g"
server_key: "1a1c87dfd0bf3a167faf0065a1f7b716"
server_key_iv: "1b72cde2d69b6140709617979c174457"
server_key_version: "encryptionkey@0"
I20260923 20:08:56.926335 3434 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790194121577027-1706-0/minicluster-data/master-0/wal/instance:
uuid: "4d74cac2e4f944208b16833320585dd4"
format_stamp: "Formatted at 2026-09-23 20:08:56 on dist-test-slave-th4g"
server_key: "1a1c87dfd0bf3a167faf0065a1f7b716"
server_key_iv: "1b72cde2d69b6140709617979c174457"
server_key_version: "encryptionkey@0"
I20260923 20:08:56.931723 3434 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260923 20:08:56.935812 3449 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:08:56.937520 3434 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260923 20:08:56.937749 3434 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790194121577027-1706-0/minicluster-data/master-0/data,/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790194121577027-1706-0/minicluster-data/master-0/wal
uuid: "4d74cac2e4f944208b16833320585dd4"
format_stamp: "Formatted at 2026-09-23 20:08:56 on dist-test-slave-th4g"
server_key: "1a1c87dfd0bf3a167faf0065a1f7b716"
server_key_iv: "1b72cde2d69b6140709617979c174457"
server_key_version: "encryptionkey@0"
I20260923 20:08:56.938022 3434 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790194121577027-1706-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790194121577027-1706-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790194121577027-1706-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:08:56.953425 3434 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:08:56.957980 3434 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:08:56.958263 3434 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:08:56.981356 3434 rpc_server.cc:307] RPC server started. Bound to: 127.1.170.190:35033
I20260923 20:08:56.981380 3510 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.170.190:35033 every 8 connection(s)
I20260923 20:08:56.982744 3434 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790194121577027-1706-0/minicluster-data/master-0/data/info.pb
I20260923 20:08:56.986542 3511 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:08:56.989773 1706 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqUuku6/build/asan/bin/kudu as pid 3434
I20260923 20:08:56.989993 1706 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790194121577027-1706-0/minicluster-data/master-0/wal/instance
I20260923 20:08:56.990793 1706 external_mini_cluster.cc:1428] Setting key 3036adf5fa95103c55852a4f8bdd9d3c
I20260923 20:08:56.999673 3511 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4d74cac2e4f944208b16833320585dd4: Bootstrap starting.
I20260923 20:08:57.003515 3511 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4d74cac2e4f944208b16833320585dd4: Neither blocks nor log segments found. Creating new log.
I20260923 20:08:57.004868 3511 log.cc:824] T 00000000000000000000000000000000 P 4d74cac2e4f944208b16833320585dd4: Log is configured to *not* fsync() on all Append() calls
Sep 23 20:08:56 dist-test-slave-th4g krb5kdc[3421](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194136, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.170.190@KRBTEST.COM
I20260923 20:08:57.008023 3511 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4d74cac2e4f944208b16833320585dd4: No bootstrap required, opened a new log
I20260923 20:08:57.014371 3514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:08:56.992890 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34700 (local address 127.1.170.190:35033)
0923 20:08:56.993702 (+ 812us) server_negotiation.cc:206] Beginning negotiation
0923 20:08:56.993716 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0923 20:08:56.993757 (+ 41us) server_negotiation.cc:407] Connection header received
0923 20:08:56.995490 (+ 1733us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:08:56.995513 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:08:56.995913 (+ 400us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:08:56.996247 (+ 334us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:08:56.998007 (+ 1760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:08:56.999287 (+ 1280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:08:57.000985 (+ 1698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:08:57.001691 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:08:57.001822 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:08:57.006871 (+ 5049us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:08:57.006879 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:08:57.006893 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:08:57.006927 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:08:57.010119 (+ 3192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:08:57.010946 (+ 827us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:08:57.010952 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:08:57.010958 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:08:57.011123 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:08:57.011507 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:08:57.011510 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:08:57.011512 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:08:57.012100 (+ 588us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:08:57.012297 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0923 20:08:57.012834 (+ 537us) server_negotiation.cc:299] Negotiation successful
0923 20:08:57.013269 (+ 435us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":591,"thread_start_us":379,"threads_started":1}
I20260923 20:08:57.016661 3511 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4d74cac2e4f944208b16833320585dd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d74cac2e4f944208b16833320585dd4" member_type: VOTER last_known_addr { host: "127.1.170.190" port: 35033 } }
I20260923 20:08:57.017042 3511 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4d74cac2e4f944208b16833320585dd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:08:57.017165 3511 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4d74cac2e4f944208b16833320585dd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d74cac2e4f944208b16833320585dd4, State: Initialized, Role: FOLLOWER
I20260923 20:08:57.017690 3511 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4d74cac2e4f944208b16833320585dd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d74cac2e4f944208b16833320585dd4" member_type: VOTER last_known_addr { host: "127.1.170.190" port: 35033 } }
I20260923 20:08:57.017964 3511 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4d74cac2e4f944208b16833320585dd4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 20:08:57.018121 3511 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4d74cac2e4f944208b16833320585dd4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 20:08:57.018283 3511 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4d74cac2e4f944208b16833320585dd4 [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:08:57.020385 3511 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4d74cac2e4f944208b16833320585dd4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d74cac2e4f944208b16833320585dd4" member_type: VOTER last_known_addr { host: "127.1.170.190" port: 35033 } }
I20260923 20:08:57.020874 3511 leader_election.cc:302] T 00000000000000000000000000000000 P 4d74cac2e4f944208b16833320585dd4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4d74cac2e4f944208b16833320585dd4; no voters:
I20260923 20:08:57.021566 3511 leader_election.cc:288] T 00000000000000000000000000000000 P 4d74cac2e4f944208b16833320585dd4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 20:08:57.021754 3516 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4d74cac2e4f944208b16833320585dd4 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 20:08:57.023113 3516 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4d74cac2e4f944208b16833320585dd4 [term 1 LEADER]: Becoming Leader. State: Replica: 4d74cac2e4f944208b16833320585dd4, State: Running, Role: LEADER
I20260923 20:08:57.023852 3511 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4d74cac2e4f944208b16833320585dd4 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 20:08:57.023641 3516 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4d74cac2e4f944208b16833320585dd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d74cac2e4f944208b16833320585dd4" member_type: VOTER last_known_addr { host: "127.1.170.190" port: 35033 } }
I20260923 20:08:57.027292 3518 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4d74cac2e4f944208b16833320585dd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4d74cac2e4f944208b16833320585dd4. Latest consensus state: current_term: 1 leader_uuid: "4d74cac2e4f944208b16833320585dd4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d74cac2e4f944208b16833320585dd4" member_type: VOTER last_known_addr { host: "127.1.170.190" port: 35033 } } }
I20260923 20:08:57.027742 3518 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4d74cac2e4f944208b16833320585dd4 [sys.catalog]: This master's current role is: LEADER
I20260923 20:08:57.028434 3517 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4d74cac2e4f944208b16833320585dd4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4d74cac2e4f944208b16833320585dd4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d74cac2e4f944208b16833320585dd4" member_type: VOTER last_known_addr { host: "127.1.170.190" port: 35033 } } }
I20260923 20:08:57.028757 3517 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4d74cac2e4f944208b16833320585dd4 [sys.catalog]: This master's current role is: LEADER
I20260923 20:08:57.029358 3525 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 20:08:57.034672 3525 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 20:08:57.043767 3525 catalog_manager.cc:1347] Generated new cluster ID: 4745bdc467c74ef18c6981ff567743df
I20260923 20:08:57.043881 3525 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 20:08:57.055370 3525 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 20:08:57.057170 3525 catalog_manager.cc:1504] Loading token signing keys...
I20260923 20:08:57.079962 3525 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4d74cac2e4f944208b16833320585dd4: Generated new TSK 0
I20260923 20:08:57.081068 3525 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.170.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.170.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.170.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790194121577027-1706-0/krb5kdc/kudu_127.1.170.129.keytab.
Entry for principal kudu/127.1.170.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790194121577027-1706-0/krb5kdc/kudu_127.1.170.129.keytab.
I20260923 20:08:57.126497 1706 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqUuku6/build/asan/bin/kudu
/tmp/dist-test-taskqUuku6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790194121577027-1706-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790194121577027-1706-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790194121577027-1706-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790194121577027-1706-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.170.129:0
--local_ip_for_outbound_sockets=127.1.170.129
--webserver_interface=127.1.170.129
--webserver_port=0
--tserver_master_addrs=127.1.170.190:35033
--keytab_file=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790194121577027-1706-0/krb5kdc/kudu_127.1.170.129.keytab
--principal=kudu/127.1.170.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.170.148:43717
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790194121577027-1706-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790194121577027-1706-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790194121577027-1706-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:08:57.457029 3537 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:08:57.457343 3537 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:08:57.457445 3537 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:08:57.465183 3537 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:08:57.465307 3537 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:08:57.465399 3537 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260923 20:08:57.465444 3537 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260923 20:08:57.465505 3537 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.170.129
I20260923 20:08:57.475394 3537 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.170.148:43717
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790194121577027-1706-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790194121577027-1706-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790194121577027-1706-0/krb5kdc/kudu_127.1.170.129.keytab
--principal=kudu/127.1.170.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.170.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790194121577027-1706-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.170.129
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.1.170.190:35033
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.170.129
--log_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790194121577027-1706-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:08:57.476676 3537 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:08:57.478008 3537 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:08:57.492892 3545 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:08:57.492970 3543 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:08:57.492928 3542 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:08:57.495653 3537 server_base.cc:1034] running on GCE node
I20260923 20:08:57.496320 3537 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:08:57.497305 3537 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:08:57.498660 3537 hybrid_clock.cc:648] HybridClock initialized: now 1790194137498606 us; error 62 us; skew 500 ppm
I20260923 20:08:57.499071 3537 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:08:57 dist-test-slave-th4g krb5kdc[3421](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194137, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.170.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:08:57.507370 3537 init.cc:377] Logged in from keytab as kudu/127.1.170.129@KRBTEST.COM (short username kudu)
I20260923 20:08:57.509063 3537 webserver.cc:460] Webserver started at http://127.1.170.129:41283/ using document root <none> and password file <none>
I20260923 20:08:57.509817 3537 fs_manager.cc:359] Metadata directory not provided
I20260923 20:08:57.509986 3537 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:08:57.510303 3537 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:08:57.513994 3537 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790194121577027-1706-0/minicluster-data/ts-0/data/instance:
uuid: "98967392b7164818a63ed7e9912395c5"
format_stamp: "Formatted at 2026-09-23 20:08:57 on dist-test-slave-th4g"
server_key: "dd21f2e11e04b818884b728607ce2a1c"
server_key_iv: "235b4fc1f8dd7c2b942bf115175a23e9"
server_key_version: "encryptionkey@0"
I20260923 20:08:57.514829 3537 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790194121577027-1706-0/minicluster-data/ts-0/wal/instance:
uuid: "98967392b7164818a63ed7e9912395c5"
format_stamp: "Formatted at 2026-09-23 20:08:57 on dist-test-slave-th4g"
server_key: "dd21f2e11e04b818884b728607ce2a1c"
server_key_iv: "235b4fc1f8dd7c2b942bf115175a23e9"
server_key_version: "encryptionkey@0"
I20260923 20:08:57.520375 3537 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260923 20:08:57.524348 3552 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:08:57.525930 3537 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260923 20:08:57.526125 3537 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790194121577027-1706-0/minicluster-data/ts-0/data,/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790194121577027-1706-0/minicluster-data/ts-0/wal
uuid: "98967392b7164818a63ed7e9912395c5"
format_stamp: "Formatted at 2026-09-23 20:08:57 on dist-test-slave-th4g"
server_key: "dd21f2e11e04b818884b728607ce2a1c"
server_key_iv: "235b4fc1f8dd7c2b942bf115175a23e9"
server_key_version: "encryptionkey@0"
I20260923 20:08:57.526398 3537 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790194121577027-1706-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790194121577027-1706-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790194121577027-1706-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:08:57.555246 3537 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:08:57.559695 3537 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:08:57.560005 3537 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:08:57.562837 3537 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 20:08:57.562976 3537 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 20:08:57.563104 3537 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 20:08:57.563198 3537 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 23 20:08:57 dist-test-slave-th4g krb5kdc[3421](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194137, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.170.129@KRBTEST.COM for kudu/127.1.170.190@KRBTEST.COM
I20260923 20:08:57.587282 3561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:08:57.563483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.170.129:49577 (local address 127.1.170.190:35033)
0923 20:08:57.565976 (+ 2493us) server_negotiation.cc:206] Beginning negotiation
0923 20:08:57.565983 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:08:57.566007 (+ 24us) server_negotiation.cc:407] Connection header received
0923 20:08:57.567828 (+ 1821us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:08:57.567834 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:08:57.567925 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:08:57.568064 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:08:57.570627 (+ 2563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:08:57.571552 (+ 925us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:08:57.574329 (+ 2777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:08:57.574855 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:08:57.574997 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:08:57.579510 (+ 4513us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:08:57.579516 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:08:57.579523 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:08:57.579572 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:08:57.583013 (+ 3441us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:08:57.584344 (+ 1331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:08:57.584349 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:08:57.584351 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:08:57.584478 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:08:57.585119 (+ 641us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:08:57.585123 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:08:57.585125 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:08:57.585474 (+ 349us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:08:57.585610 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0923 20:08:57.586757 (+ 1147us) server_negotiation.cc:299] Negotiation successful
0923 20:08:57.587020 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2326,"thread_start_us":2212,"threads_started":1}
I20260923 20:08:57.590082 3562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:08:57.564222 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.190:35033 (local address 127.1.170.129:49577)
0923 20:08:57.565155 (+ 933us) negotiation.cc:107] Waiting for socket to connect
0923 20:08:57.565202 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0923 20:08:57.567243 (+ 2041us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:08:57.568481 (+ 1238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:08:57.568496 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:08:57.569362 (+ 866us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:08:57.570425 (+ 1063us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:08:57.570451 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:08:57.571733 (+ 1282us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:08:57.571741 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:08:57.574167 (+ 2426us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:08:57.574180 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:08:57.575100 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:08:57.575110 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:08:57.575361 (+ 251us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:08:57.576092 (+ 731us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:08:57.576116 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:08:57.579121 (+ 3005us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:08:57.583184 (+ 4063us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:08:57.583199 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:08:57.583226 (+ 27us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:08:57.584123 (+ 897us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:08:57.584600 (+ 477us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:08:57.584607 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:08:57.584613 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:08:57.584942 (+ 329us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:08:57.585620 (+ 678us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:08:57.585633 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:08:57.586277 (+ 644us) client_negotiation.cc:769] Sending connection context
0923 20:08:57.586659 (+ 382us) client_negotiation.cc:240] Negotiation successful
0923 20:08:57.588012 (+ 1353us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":661,"thread_start_us":430,"threads_started":1}
I20260923 20:08:57.611402 3624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:08:57.596823 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.190:35033 (local address 127.1.170.129:56191)
0923 20:08:57.597736 (+ 913us) negotiation.cc:107] Waiting for socket to connect
0923 20:08:57.597761 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0923 20:08:57.597982 (+ 221us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:08:57.598347 (+ 365us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:08:57.598355 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:08:57.598857 (+ 502us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:08:57.599170 (+ 313us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:08:57.599180 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:08:57.600204 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:08:57.600210 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:08:57.601309 (+ 1099us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:08:57.601320 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:08:57.602146 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:08:57.602156 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:08:57.602349 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:08:57.603095 (+ 746us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:08:57.603126 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:08:57.604644 (+ 1518us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:08:57.608411 (+ 3767us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:08:57.608417 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:08:57.608421 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:08:57.609099 (+ 678us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:08:57.609514 (+ 415us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:08:57.609519 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:08:57.609523 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:08:57.609667 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:08:57.610281 (+ 614us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:08:57.610286 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:08:57.610556 (+ 270us) client_negotiation.cc:769] Sending connection context
0923 20:08:57.610760 (+ 204us) client_negotiation.cc:240] Negotiation successful
0923 20:08:57.611072 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":697,"thread_start_us":514,"threads_started":1}
I20260923 20:08:57.611757 3561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:08:57.596708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.170.129:56191 (local address 127.1.170.190:35033)
0923 20:08:57.596905 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0923 20:08:57.596912 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:08:57.597872 (+ 960us) server_negotiation.cc:407] Connection header received
0923 20:08:57.598134 (+ 262us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:08:57.598138 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:08:57.598210 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:08:57.598317 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:08:57.599325 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:08:57.600054 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:08:57.601469 (+ 1415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:08:57.601941 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:08:57.602084 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:08:57.604889 (+ 2805us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:08:57.604899 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:08:57.604905 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:08:57.604952 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:08:57.608198 (+ 3246us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:08:57.609254 (+ 1056us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:08:57.609262 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:08:57.609266 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:08:57.609394 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:08:57.609785 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:08:57.609792 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:08:57.609797 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:08:57.610158 (+ 361us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:08:57.610312 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0923 20:08:57.610939 (+ 627us) server_negotiation.cc:299] Negotiation successful
0923 20:08:57.611259 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260923 20:08:57.623759 3537 rpc_server.cc:307] RPC server started. Bound to: 127.1.170.129:44623
I20260923 20:08:57.623797 3672 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.170.129:44623 every 8 connection(s)
I20260923 20:08:57.625388 3537 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790194121577027-1706-0/minicluster-data/ts-0/data/info.pb
I20260923 20:08:57.626073 1706 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqUuku6/build/asan/bin/kudu as pid 3537
I20260923 20:08:57.626288 1706 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790194121577027-1706-0/minicluster-data/ts-0/wal/instance
I20260923 20:08:57.627295 1706 external_mini_cluster.cc:1428] Setting key f70bd8cb342e9232a26158ac2de40036
I20260923 20:08:57.627713 3673 heartbeater.cc:347] Connected to a master server at 127.1.170.190:35033
I20260923 20:08:57.627914 3673 heartbeater.cc:464] Registering TS with master...
I20260923 20:08:57.628710 3673 heartbeater.cc:511] Master 127.1.170.190:35033 requested a full tablet report, sending...
I20260923 20:08:57.631271 3466 ts_manager.cc:194] Registered new tserver with Master: 98967392b7164818a63ed7e9912395c5 (127.1.170.129:44623)
I20260923 20:08:57.632826 3466 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.170.129@KRBTEST.COM'} at 127.1.170.129:49577
WARNING: no policy specified for kudu/127.1.170.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.170.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.170.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790194121577027-1706-0/krb5kdc/kudu_127.1.170.130.keytab.
Entry for principal kudu/127.1.170.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790194121577027-1706-0/krb5kdc/kudu_127.1.170.130.keytab.
I20260923 20:08:57.674237 1706 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqUuku6/build/asan/bin/kudu
/tmp/dist-test-taskqUuku6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790194121577027-1706-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790194121577027-1706-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790194121577027-1706-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790194121577027-1706-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.170.130:0
--local_ip_for_outbound_sockets=127.1.170.130
--webserver_interface=127.1.170.130
--webserver_port=0
--tserver_master_addrs=127.1.170.190:35033
--keytab_file=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790194121577027-1706-0/krb5kdc/kudu_127.1.170.130.keytab
--principal=kudu/127.1.170.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.170.148:43717
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790194121577027-1706-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790194121577027-1706-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790194121577027-1706-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:08:58.021657 3677 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:08:58.022030 3677 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:08:58.022158 3677 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:08:58.029489 3677 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:08:58.029611 3677 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:08:58.029739 3677 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260923 20:08:58.029815 3677 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260923 20:08:58.029917 3677 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.170.130
I20260923 20:08:58.040666 3677 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.170.148:43717
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790194121577027-1706-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790194121577027-1706-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790194121577027-1706-0/krb5kdc/kudu_127.1.170.130.keytab
--principal=kudu/127.1.170.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.170.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790194121577027-1706-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.170.130
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.1.170.190:35033
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.170.130
--log_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790194121577027-1706-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:08:58.042136 3677 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:08:58.043644 3677 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:08:58.058393 3682 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:08:58.058450 3685 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:08:58.058430 3683 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:08:58.059147 3677 server_base.cc:1034] running on GCE node
I20260923 20:08:58.059823 3677 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:08:58.060894 3677 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:08:58.062139 3677 hybrid_clock.cc:648] HybridClock initialized: now 1790194138062048 us; error 86 us; skew 500 ppm
I20260923 20:08:58.062525 3677 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:08:58 dist-test-slave-th4g krb5kdc[3421](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194138, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.170.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:08:58.066196 3677 init.cc:377] Logged in from keytab as kudu/127.1.170.130@KRBTEST.COM (short username kudu)
I20260923 20:08:58.068030 3677 webserver.cc:460] Webserver started at http://127.1.170.130:44553/ using document root <none> and password file <none>
I20260923 20:08:58.068758 3677 fs_manager.cc:359] Metadata directory not provided
I20260923 20:08:58.068908 3677 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:08:58.069224 3677 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:08:58.072582 3677 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790194121577027-1706-0/minicluster-data/ts-1/data/instance:
uuid: "df7bc6713ea546638a209a6b211dfa0c"
format_stamp: "Formatted at 2026-09-23 20:08:58 on dist-test-slave-th4g"
server_key: "08fce5ca57cde74fae21c6436c86df82"
server_key_iv: "91366ca350d1502b3a915fce5acfd35e"
server_key_version: "encryptionkey@0"
I20260923 20:08:58.073511 3677 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790194121577027-1706-0/minicluster-data/ts-1/wal/instance:
uuid: "df7bc6713ea546638a209a6b211dfa0c"
format_stamp: "Formatted at 2026-09-23 20:08:58 on dist-test-slave-th4g"
server_key: "08fce5ca57cde74fae21c6436c86df82"
server_key_iv: "91366ca350d1502b3a915fce5acfd35e"
server_key_version: "encryptionkey@0"
I20260923 20:08:58.079713 3677 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.003s
I20260923 20:08:58.084856 3692 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:08:58.087469 3677 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.000s
I20260923 20:08:58.087687 3677 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790194121577027-1706-0/minicluster-data/ts-1/data,/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790194121577027-1706-0/minicluster-data/ts-1/wal
uuid: "df7bc6713ea546638a209a6b211dfa0c"
format_stamp: "Formatted at 2026-09-23 20:08:58 on dist-test-slave-th4g"
server_key: "08fce5ca57cde74fae21c6436c86df82"
server_key_iv: "91366ca350d1502b3a915fce5acfd35e"
server_key_version: "encryptionkey@0"
I20260923 20:08:58.087909 3677 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790194121577027-1706-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790194121577027-1706-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790194121577027-1706-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:08:58.102996 3677 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:08:58.107836 3677 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:08:58.108191 3677 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:08:58.111469 3677 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 20:08:58.111572 3677 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 20:08:58.111720 3677 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 20:08:58.111815 3677 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 23 20:08:58 dist-test-slave-th4g krb5kdc[3421](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194138, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.170.130@KRBTEST.COM for kudu/127.1.170.190@KRBTEST.COM
I20260923 20:08:58.139415 3561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:08:58.112014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.170.130:59263 (local address 127.1.170.190:35033)
0923 20:08:58.112465 (+ 451us) server_negotiation.cc:206] Beginning negotiation
0923 20:08:58.112473 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 20:08:58.114671 (+ 2198us) server_negotiation.cc:407] Connection header received
0923 20:08:58.117441 (+ 2770us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:08:58.117451 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:08:58.117557 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:08:58.117619 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:08:58.120354 (+ 2735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:08:58.121685 (+ 1331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:08:58.124678 (+ 2993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:08:58.125297 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:08:58.125453 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:08:58.132113 (+ 6660us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:08:58.132123 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:08:58.132131 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:08:58.132191 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:08:58.135940 (+ 3749us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:08:58.136851 (+ 911us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:08:58.136857 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:08:58.136861 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:08:58.136985 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:08:58.137449 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:08:58.137455 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:08:58.137460 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:08:58.137748 (+ 288us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:08:58.137955 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0923 20:08:58.138793 (+ 838us) server_negotiation.cc:299] Negotiation successful
0923 20:08:58.139042 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":269}
I20260923 20:08:58.141378 3701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:08:58.113136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.190:35033 (local address 127.1.170.130:59263)
0923 20:08:58.114436 (+ 1300us) negotiation.cc:107] Waiting for socket to connect
0923 20:08:58.114498 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0923 20:08:58.117040 (+ 2542us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:08:58.118226 (+ 1186us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:08:58.118238 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:08:58.119220 (+ 982us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:08:58.120061 (+ 841us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:08:58.120100 (+ 39us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:08:58.121969 (+ 1869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:08:58.121974 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:08:58.124380 (+ 2406us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:08:58.124395 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:08:58.125714 (+ 1319us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:08:58.125727 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:08:58.126186 (+ 459us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:08:58.127197 (+ 1011us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:08:58.127243 (+ 46us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:08:58.131740 (+ 4497us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:08:58.136156 (+ 4416us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:08:58.136167 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:08:58.136190 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:08:58.136695 (+ 505us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:08:58.137111 (+ 416us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:08:58.137116 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:08:58.137120 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:08:58.137327 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:08:58.137933 (+ 606us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:08:58.137942 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:08:58.138375 (+ 433us) client_negotiation.cc:769] Sending connection context
0923 20:08:58.138703 (+ 328us) client_negotiation.cc:240] Negotiation successful
0923 20:08:58.139347 (+ 644us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":967,"thread_start_us":707,"threads_started":1}
I20260923 20:08:58.163342 3677 rpc_server.cc:307] RPC server started. Bound to: 127.1.170.130:39109
I20260923 20:08:58.164865 3561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:08:58.148331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.170.130:53781 (local address 127.1.170.190:35033)
0923 20:08:58.148533 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0923 20:08:58.148540 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:08:58.149377 (+ 837us) server_negotiation.cc:407] Connection header received
0923 20:08:58.149573 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:08:58.149582 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:08:58.149706 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:08:58.149972 (+ 266us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:08:58.151026 (+ 1054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:08:58.152556 (+ 1530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:08:58.154059 (+ 1503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:08:58.154498 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:08:58.154651 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:08:58.157565 (+ 2914us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:08:58.157572 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:08:58.157579 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:08:58.157630 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:08:58.161229 (+ 3599us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:08:58.162264 (+ 1035us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:08:58.162270 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:08:58.162273 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:08:58.162444 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:08:58.162883 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:08:58.162890 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:08:58.162894 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:08:58.163233 (+ 339us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:08:58.163396 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0923 20:08:58.163981 (+ 585us) server_negotiation.cc:299] Negotiation successful
0923 20:08:58.164497 (+ 516us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260923 20:08:58.163367 3811 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.170.130:39109 every 8 connection(s)
I20260923 20:08:58.165983 3787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:08:58.148366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.190:35033 (local address 127.1.170.130:53781)
0923 20:08:58.149206 (+ 840us) negotiation.cc:107] Waiting for socket to connect
0923 20:08:58.149255 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0923 20:08:58.149403 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:08:58.150010 (+ 607us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:08:58.150014 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:08:58.150522 (+ 508us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:08:58.150857 (+ 335us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:08:58.150867 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:08:58.152749 (+ 1882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:08:58.152754 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:08:58.153826 (+ 1072us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:08:58.153838 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:08:58.154704 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:08:58.154710 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:08:58.154902 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:08:58.155682 (+ 780us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:08:58.155717 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:08:58.157189 (+ 1472us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:08:58.161384 (+ 4195us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:08:58.161389 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:08:58.161393 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:08:58.162073 (+ 680us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:08:58.162611 (+ 538us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:08:58.162615 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:08:58.162618 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:08:58.162750 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:08:58.163364 (+ 614us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:08:58.163369 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:08:58.163620 (+ 251us) client_negotiation.cc:769] Sending connection context
0923 20:08:58.163814 (+ 194us) client_negotiation.cc:240] Negotiation successful
0923 20:08:58.164260 (+ 446us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":699,"thread_start_us":559,"threads_started":1}
I20260923 20:08:58.165009 3677 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790194121577027-1706-0/minicluster-data/ts-1/data/info.pb
I20260923 20:08:58.168779 3812 heartbeater.cc:347] Connected to a master server at 127.1.170.190:35033
I20260923 20:08:58.169276 3812 heartbeater.cc:464] Registering TS with master...
I20260923 20:08:58.169991 3812 heartbeater.cc:511] Master 127.1.170.190:35033 requested a full tablet report, sending...
I20260923 20:08:58.171554 3466 ts_manager.cc:194] Registered new tserver with Master: df7bc6713ea546638a209a6b211dfa0c (127.1.170.130:39109)
I20260923 20:08:58.172441 3466 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.170.130@KRBTEST.COM'} at 127.1.170.130:59263
I20260923 20:08:58.175356 1706 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqUuku6/build/asan/bin/kudu as pid 3677
I20260923 20:08:58.175570 1706 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790194121577027-1706-0/minicluster-data/ts-1/wal/instance
I20260923 20:08:58.176017 1706 external_mini_cluster.cc:1428] Setting key 22d6cfe07de7cd65840bec6946acf5a8
WARNING: no policy specified for kudu/127.1.170.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.170.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.170.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790194121577027-1706-0/krb5kdc/kudu_127.1.170.131.keytab.
Entry for principal kudu/127.1.170.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790194121577027-1706-0/krb5kdc/kudu_127.1.170.131.keytab.
I20260923 20:08:58.223263 1706 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqUuku6/build/asan/bin/kudu
/tmp/dist-test-taskqUuku6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790194121577027-1706-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790194121577027-1706-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790194121577027-1706-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790194121577027-1706-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.170.131:0
--local_ip_for_outbound_sockets=127.1.170.131
--webserver_interface=127.1.170.131
--webserver_port=0
--tserver_master_addrs=127.1.170.190:35033
--keytab_file=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790194121577027-1706-0/krb5kdc/kudu_127.1.170.131.keytab
--principal=kudu/127.1.170.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.170.148:43717
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790194121577027-1706-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790194121577027-1706-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790194121577027-1706-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:08:58.568650 3816 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:08:58.568972 3816 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:08:58.569136 3816 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:08:58.579829 3816 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:08:58.579988 3816 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:08:58.580142 3816 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260923 20:08:58.580222 3816 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260923 20:08:58.580332 3816 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.170.131
I20260923 20:08:58.592278 3816 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.170.148:43717
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790194121577027-1706-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790194121577027-1706-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790194121577027-1706-0/krb5kdc/kudu_127.1.170.131.keytab
--principal=kudu/127.1.170.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.170.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790194121577027-1706-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.170.131
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.1.170.190:35033
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.170.131
--log_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790194121577027-1706-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:08:58.593756 3816 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:08:58.595165 3816 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:08:58.610489 3821 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:08:58.610555 3824 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:08:58.610519 3822 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:08:58.611619 3816 server_base.cc:1034] running on GCE node
I20260923 20:08:58.612310 3816 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:08:58.613356 3816 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:08:58.614660 3816 hybrid_clock.cc:648] HybridClock initialized: now 1790194138614604 us; error 41 us; skew 500 ppm
I20260923 20:08:58.615072 3816 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:08:58 dist-test-slave-th4g krb5kdc[3421](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194138, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.170.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:08:58.619532 3816 init.cc:377] Logged in from keytab as kudu/127.1.170.131@KRBTEST.COM (short username kudu)
I20260923 20:08:58.621536 3816 webserver.cc:460] Webserver started at http://127.1.170.131:44741/ using document root <none> and password file <none>
I20260923 20:08:58.622546 3816 fs_manager.cc:359] Metadata directory not provided
I20260923 20:08:58.622710 3816 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:08:58.623052 3816 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:08:58.626451 3816 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790194121577027-1706-0/minicluster-data/ts-2/data/instance:
uuid: "659357c7635644ea8fad67cfbd3c3c22"
format_stamp: "Formatted at 2026-09-23 20:08:58 on dist-test-slave-th4g"
server_key: "c2419a3c94c85e642d8285dc29b79ea9"
server_key_iv: "566015090393e122428d37c1c98a94c3"
server_key_version: "encryptionkey@0"
I20260923 20:08:58.627256 3816 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790194121577027-1706-0/minicluster-data/ts-2/wal/instance:
uuid: "659357c7635644ea8fad67cfbd3c3c22"
format_stamp: "Formatted at 2026-09-23 20:08:58 on dist-test-slave-th4g"
server_key: "c2419a3c94c85e642d8285dc29b79ea9"
server_key_iv: "566015090393e122428d37c1c98a94c3"
server_key_version: "encryptionkey@0"
I20260923 20:08:58.633270 3816 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260923 20:08:58.636770 3673 heartbeater.cc:503] Master 127.1.170.190:35033 was elected leader, sending a full tablet report...
I20260923 20:08:58.638072 3831 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:08:58.640194 3816 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260923 20:08:58.640415 3816 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790194121577027-1706-0/minicluster-data/ts-2/data,/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790194121577027-1706-0/minicluster-data/ts-2/wal
uuid: "659357c7635644ea8fad67cfbd3c3c22"
format_stamp: "Formatted at 2026-09-23 20:08:58 on dist-test-slave-th4g"
server_key: "c2419a3c94c85e642d8285dc29b79ea9"
server_key_iv: "566015090393e122428d37c1c98a94c3"
server_key_version: "encryptionkey@0"
I20260923 20:08:58.640641 3816 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790194121577027-1706-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790194121577027-1706-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790194121577027-1706-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:08:58.678467 3816 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:08:58.683509 3816 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:08:58.683840 3816 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:08:58.687011 3816 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 20:08:58.687157 3816 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 20:08:58.687305 3816 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 20:08:58.687413 3816 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 23 20:08:58 dist-test-slave-th4g krb5kdc[3421](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194138, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.170.131@KRBTEST.COM for kudu/127.1.170.190@KRBTEST.COM
I20260923 20:08:58.715107 3840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:08:58.687631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.170.131:60017 (local address 127.1.170.190:35033)
0923 20:08:58.688337 (+ 706us) server_negotiation.cc:206] Beginning negotiation
0923 20:08:58.688344 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:08:58.689454 (+ 1110us) server_negotiation.cc:407] Connection header received
0923 20:08:58.692585 (+ 3131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:08:58.692591 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:08:58.692686 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:08:58.692843 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:08:58.695425 (+ 2582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:08:58.696757 (+ 1332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:08:58.699394 (+ 2637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:08:58.700074 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:08:58.700254 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:08:58.705960 (+ 5706us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:08:58.705971 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:08:58.705979 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:08:58.706042 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:08:58.709632 (+ 3590us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:08:58.711531 (+ 1899us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:08:58.711539 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:08:58.711542 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:08:58.711788 (+ 246us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:08:58.712520 (+ 732us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:08:58.712526 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:08:58.712529 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:08:58.712892 (+ 363us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:08:58.713035 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0923 20:08:58.714431 (+ 1396us) server_negotiation.cc:299] Negotiation successful
0923 20:08:58.714675 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":574,"thread_start_us":475,"threads_started":1}
I20260923 20:08:58.717098 3841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:08:58.688371 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.190:35033 (local address 127.1.170.131:60017)
0923 20:08:58.689246 (+ 875us) negotiation.cc:107] Waiting for socket to connect
0923 20:08:58.689305 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0923 20:08:58.692173 (+ 2868us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:08:58.693194 (+ 1021us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:08:58.693207 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:08:58.694151 (+ 944us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:08:58.695148 (+ 997us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:08:58.695196 (+ 48us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:08:58.696982 (+ 1786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:08:58.696990 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:08:58.699214 (+ 2224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:08:58.699232 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:08:58.700418 (+ 1186us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:08:58.700434 (+ 16us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:08:58.700834 (+ 400us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:08:58.701904 (+ 1070us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:08:58.701957 (+ 53us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:08:58.705556 (+ 3599us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:08:58.709831 (+ 4275us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:08:58.709845 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:08:58.709898 (+ 53us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:08:58.711209 (+ 1311us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:08:58.711958 (+ 749us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:08:58.711966 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:08:58.711973 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:08:58.712307 (+ 334us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:08:58.713113 (+ 806us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:08:58.713127 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:08:58.713820 (+ 693us) client_negotiation.cc:769] Sending connection context
0923 20:08:58.714261 (+ 441us) client_negotiation.cc:240] Negotiation successful
0923 20:08:58.715066 (+ 805us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":583,"thread_start_us":375,"threads_started":1}
I20260923 20:08:58.743073 3816 rpc_server.cc:307] RPC server started. Bound to: 127.1.170.131:37945
I20260923 20:08:58.743106 3951 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.170.131:37945 every 8 connection(s)
I20260923 20:08:58.743384 3840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:08:58.724316 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.170.131:42365 (local address 127.1.170.190:35033)
0923 20:08:58.724539 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0923 20:08:58.724548 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 20:08:58.725628 (+ 1080us) server_negotiation.cc:407] Connection header received
0923 20:08:58.725991 (+ 363us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:08:58.726000 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:08:58.726109 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:08:58.726258 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:08:58.727704 (+ 1446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:08:58.728817 (+ 1113us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:08:58.730567 (+ 1750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:08:58.731207 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:08:58.731368 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:08:58.734486 (+ 3118us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:08:58.734497 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:08:58.734505 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:08:58.734561 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:08:58.738985 (+ 4424us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:08:58.740043 (+ 1058us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:08:58.740053 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:08:58.740059 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:08:58.740245 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:08:58.740750 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:08:58.740755 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:08:58.740758 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:08:58.741173 (+ 415us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:08:58.741378 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0923 20:08:58.742598 (+ 1220us) server_negotiation.cc:299] Negotiation successful
0923 20:08:58.742911 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260923 20:08:58.743777 3915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:08:58.724510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.190:35033 (local address 127.1.170.131:42365)
0923 20:08:58.725463 (+ 953us) negotiation.cc:107] Waiting for socket to connect
0923 20:08:58.725492 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0923 20:08:58.725769 (+ 277us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:08:58.726288 (+ 519us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:08:58.726293 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:08:58.726992 (+ 699us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:08:58.727541 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:08:58.727552 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:08:58.728962 (+ 1410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:08:58.728967 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:08:58.730361 (+ 1394us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:08:58.730373 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:08:58.731745 (+ 1372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:08:58.731754 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:08:58.731989 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:08:58.732820 (+ 831us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:08:58.732851 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:08:58.734197 (+ 1346us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:08:58.739186 (+ 4989us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:08:58.739204 (+ 18us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:08:58.739210 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:08:58.739851 (+ 641us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:08:58.740408 (+ 557us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:08:58.740415 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:08:58.740421 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:08:58.740616 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:08:58.741486 (+ 870us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:08:58.741495 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:08:58.742005 (+ 510us) client_negotiation.cc:769] Sending connection context
0923 20:08:58.742266 (+ 261us) client_negotiation.cc:240] Negotiation successful
0923 20:08:58.742864 (+ 598us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":666,"thread_start_us":625,"threads_started":1}
I20260923 20:08:58.744927 3816 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790194121577027-1706-0/minicluster-data/ts-2/data/info.pb
I20260923 20:08:58.745944 1706 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqUuku6/build/asan/bin/kudu as pid 3816
I20260923 20:08:58.746209 1706 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790194121577027-1706-0/minicluster-data/ts-2/wal/instance
I20260923 20:08:58.747031 1706 external_mini_cluster.cc:1428] Setting key e86bb016bee2744e07a8aff6039db483
I20260923 20:08:58.748591 3952 heartbeater.cc:347] Connected to a master server at 127.1.170.190:35033
I20260923 20:08:58.748871 3952 heartbeater.cc:464] Registering TS with master...
I20260923 20:08:58.749733 3952 heartbeater.cc:511] Master 127.1.170.190:35033 requested a full tablet report, sending...
I20260923 20:08:58.751327 3465 ts_manager.cc:194] Registered new tserver with Master: 659357c7635644ea8fad67cfbd3c3c22 (127.1.170.131:37945)
I20260923 20:08:58.752238 3465 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.170.131@KRBTEST.COM'} at 127.1.170.131:60017
I20260923 20:08:58.763532 1706 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 23 20:08:58 dist-test-slave-th4g krb5kdc[3421](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194138, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 23 20:08:58 dist-test-slave-th4g krb5kdc[3421](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194138, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.170.190@KRBTEST.COM
I20260923 20:08:58.818413 3840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:08:58.800947 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34708 (local address 127.1.170.190:35033)
0923 20:08:58.801206 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0923 20:08:58.801213 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:08:58.801594 (+ 381us) server_negotiation.cc:407] Connection header received
0923 20:08:58.801866 (+ 272us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:08:58.801870 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:08:58.801995 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:08:58.802106 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:08:58.803822 (+ 1716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:08:58.805000 (+ 1178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:08:58.806524 (+ 1524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:08:58.806993 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:08:58.807134 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:08:58.811910 (+ 4776us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:08:58.811915 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:08:58.811919 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:08:58.811961 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:08:58.814924 (+ 2963us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:08:58.816290 (+ 1366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:08:58.816295 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:08:58.816298 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:08:58.816449 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:08:58.816870 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:08:58.816877 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:08:58.816880 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:08:58.817167 (+ 287us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:08:58.817474 (+ 307us) server_negotiation.cc:1035] Waiting for connection context
0923 20:08:58.817751 (+ 277us) server_negotiation.cc:299] Negotiation successful
0923 20:08:58.818030 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101}
I20260923 20:08:58.823604 3465 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34708:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 20:08:58.831282 3465 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 20:08:58.867415 3969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:08:58.850412 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.129:44623 (local address 127.0.0.1:38920)
0923 20:08:58.853866 (+ 3454us) negotiation.cc:107] Waiting for socket to connect
0923 20:08:58.853948 (+ 82us) client_negotiation.cc:174] Beginning negotiation
0923 20:08:58.854294 (+ 346us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:08:58.856214 (+ 1920us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:08:58.856226 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:08:58.856307 (+ 81us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:08:58.856892 (+ 585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:08:58.856911 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:08:58.859604 (+ 2693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:08:58.859613 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:08:58.863186 (+ 3573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:08:58.863211 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:08:58.866108 (+ 2897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:08:58.866114 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:08:58.866413 (+ 299us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:08:58.866478 (+ 65us) client_negotiation.cc:769] Sending connection context
0923 20:08:58.866649 (+ 171us) client_negotiation.cc:240] Negotiation successful
0923 20:08:58.866875 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3215,"thread_start_us":1412,"threads_started":1}
I20260923 20:08:58.867527 3968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:08:58.851590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38920 (local address 127.1.170.129:44623)
0923 20:08:58.852936 (+ 1346us) server_negotiation.cc:206] Beginning negotiation
0923 20:08:58.852949 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0923 20:08:58.854109 (+ 1160us) server_negotiation.cc:407] Connection header received
0923 20:08:58.854590 (+ 481us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:08:58.854608 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:08:58.855275 (+ 667us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:08:58.855757 (+ 482us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:08:58.857171 (+ 1414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:08:58.859285 (+ 2114us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:08:58.863505 (+ 4220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:08:58.865632 (+ 2127us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:08:58.865767 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:08:58.866024 (+ 257us) server_negotiation.cc:1035] Waiting for connection context
0923 20:08:58.866828 (+ 804us) server_negotiation.cc:299] Negotiation successful
0923 20:08:58.867138 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1121,"thread_start_us":2339,"threads_started":1}
I20260923 20:08:58.872830 3603 tablet_service.cc:1467] Processing CreateTablet for tablet 906c58e7c4714f7bad7b4f568bfcebd7 (DEFAULT_TABLE table=test-table [id=4ad717d57bde40b3a63886d3cf533a37]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 20:08:58.874596 3965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:08:58.852993 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.130:39109 (local address 127.0.0.1:38888)
0923 20:08:58.865225 (+ 12232us) negotiation.cc:107] Waiting for socket to connect
0923 20:08:58.865249 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0923 20:08:58.865622 (+ 373us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:08:58.866754 (+ 1132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:08:58.866758 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:08:58.866796 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:08:58.867254 (+ 458us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:08:58.867266 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:08:58.869465 (+ 2199us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:08:58.869473 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:08:58.871344 (+ 1871us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:08:58.871363 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:08:58.873448 (+ 2085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:08:58.873455 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:08:58.873826 (+ 371us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:08:58.873869 (+ 43us) client_negotiation.cc:769] Sending connection context
0923 20:08:58.874019 (+ 150us) client_negotiation.cc:240] Negotiation successful
0923 20:08:58.874178 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12082,"thread_start_us":1229,"threads_started":1}
I20260923 20:08:58.875053 3603 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 906c58e7c4714f7bad7b4f568bfcebd7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:08:58.875463 3970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:08:58.853664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38888 (local address 127.1.170.130:39109)
0923 20:08:58.864388 (+ 10724us) server_negotiation.cc:206] Beginning negotiation
0923 20:08:58.864403 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0923 20:08:58.865445 (+ 1042us) server_negotiation.cc:407] Connection header received
0923 20:08:58.865835 (+ 390us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:08:58.865844 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:08:58.866212 (+ 368us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:08:58.866446 (+ 234us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:08:58.867509 (+ 1063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:08:58.869284 (+ 1775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:08:58.871528 (+ 2244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:08:58.872901 (+ 1373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:08:58.873052 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:08:58.873284 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0923 20:08:58.874982 (+ 1698us) server_negotiation.cc:299] Negotiation successful
0923 20:08:58.875190 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10337,"thread_start_us":10750,"threads_started":1}
I20260923 20:08:58.876497 3967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:08:58.849769 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.131:37945 (local address 127.0.0.1:33456)
0923 20:08:58.851695 (+ 1926us) negotiation.cc:107] Waiting for socket to connect
0923 20:08:58.851785 (+ 90us) client_negotiation.cc:174] Beginning negotiation
0923 20:08:58.852105 (+ 320us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:08:58.867549 (+ 15444us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:08:58.867557 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:08:58.867601 (+ 44us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:08:58.868059 (+ 458us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:08:58.868071 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:08:58.871129 (+ 3058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:08:58.871135 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:08:58.873155 (+ 2020us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:08:58.873184 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:08:58.875496 (+ 2312us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:08:58.875504 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:08:58.875723 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:08:58.875770 (+ 47us) client_negotiation.cc:769] Sending connection context
0923 20:08:58.875880 (+ 110us) client_negotiation.cc:240] Negotiation successful
0923 20:08:58.876064 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1493,"thread_start_us":15149,"threads_started":1}
I20260923 20:08:58.876686 3966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:08:58.850168 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33456 (local address 127.1.170.131:37945)
0923 20:08:58.865683 (+ 15515us) server_negotiation.cc:206] Beginning negotiation
0923 20:08:58.865692 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 20:08:58.865726 (+ 34us) server_negotiation.cc:407] Connection header received
0923 20:08:58.865826 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:08:58.865835 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:08:58.866673 (+ 838us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:08:58.868497 (+ 1824us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:08:58.868856 (+ 359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:08:58.870941 (+ 2085us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:08:58.873496 (+ 2555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:08:58.875226 (+ 1730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:08:58.875421 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:08:58.875722 (+ 301us) server_negotiation.cc:1035] Waiting for connection context
0923 20:08:58.876050 (+ 328us) server_negotiation.cc:299] Negotiation successful
0923 20:08:58.876263 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15316,"thread_start_us":16914,"threads_started":1}
I20260923 20:08:58.880219 3742 tablet_service.cc:1467] Processing CreateTablet for tablet 906c58e7c4714f7bad7b4f568bfcebd7 (DEFAULT_TABLE table=test-table [id=4ad717d57bde40b3a63886d3cf533a37]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 20:08:58.881126 3882 tablet_service.cc:1467] Processing CreateTablet for tablet 906c58e7c4714f7bad7b4f568bfcebd7 (DEFAULT_TABLE table=test-table [id=4ad717d57bde40b3a63886d3cf533a37]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 20:08:58.881989 3742 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 906c58e7c4714f7bad7b4f568bfcebd7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:08:58.882397 3882 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 906c58e7c4714f7bad7b4f568bfcebd7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:08:58.893483 3971 tablet_bootstrap.cc:492] T 906c58e7c4714f7bad7b4f568bfcebd7 P 98967392b7164818a63ed7e9912395c5: Bootstrap starting.
I20260923 20:08:58.898443 3971 tablet_bootstrap.cc:654] T 906c58e7c4714f7bad7b4f568bfcebd7 P 98967392b7164818a63ed7e9912395c5: Neither blocks nor log segments found. Creating new log.
I20260923 20:08:58.900398 3973 tablet_bootstrap.cc:492] T 906c58e7c4714f7bad7b4f568bfcebd7 P df7bc6713ea546638a209a6b211dfa0c: Bootstrap starting.
I20260923 20:08:58.902504 3971 log.cc:824] T 906c58e7c4714f7bad7b4f568bfcebd7 P 98967392b7164818a63ed7e9912395c5: Log is configured to *not* fsync() on all Append() calls
I20260923 20:08:58.902710 3972 tablet_bootstrap.cc:492] T 906c58e7c4714f7bad7b4f568bfcebd7 P 659357c7635644ea8fad67cfbd3c3c22: Bootstrap starting.
I20260923 20:08:58.904728 3973 tablet_bootstrap.cc:654] T 906c58e7c4714f7bad7b4f568bfcebd7 P df7bc6713ea546638a209a6b211dfa0c: Neither blocks nor log segments found. Creating new log.
I20260923 20:08:58.906085 3973 log.cc:824] T 906c58e7c4714f7bad7b4f568bfcebd7 P df7bc6713ea546638a209a6b211dfa0c: Log is configured to *not* fsync() on all Append() calls
I20260923 20:08:58.906203 3971 tablet_bootstrap.cc:492] T 906c58e7c4714f7bad7b4f568bfcebd7 P 98967392b7164818a63ed7e9912395c5: No bootstrap required, opened a new log
I20260923 20:08:58.906314 3972 tablet_bootstrap.cc:654] T 906c58e7c4714f7bad7b4f568bfcebd7 P 659357c7635644ea8fad67cfbd3c3c22: Neither blocks nor log segments found. Creating new log.
I20260923 20:08:58.906553 3971 ts_tablet_manager.cc:1397] T 906c58e7c4714f7bad7b4f568bfcebd7 P 98967392b7164818a63ed7e9912395c5: Time spent bootstrapping tablet: real 0.013s user 0.007s sys 0.003s
I20260923 20:08:58.907442 3972 log.cc:824] T 906c58e7c4714f7bad7b4f568bfcebd7 P 659357c7635644ea8fad67cfbd3c3c22: Log is configured to *not* fsync() on all Append() calls
I20260923 20:08:58.909554 3973 tablet_bootstrap.cc:492] T 906c58e7c4714f7bad7b4f568bfcebd7 P df7bc6713ea546638a209a6b211dfa0c: No bootstrap required, opened a new log
I20260923 20:08:58.909857 3973 ts_tablet_manager.cc:1397] T 906c58e7c4714f7bad7b4f568bfcebd7 P df7bc6713ea546638a209a6b211dfa0c: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260923 20:08:58.910466 3972 tablet_bootstrap.cc:492] T 906c58e7c4714f7bad7b4f568bfcebd7 P 659357c7635644ea8fad67cfbd3c3c22: No bootstrap required, opened a new log
I20260923 20:08:58.910722 3972 ts_tablet_manager.cc:1397] T 906c58e7c4714f7bad7b4f568bfcebd7 P 659357c7635644ea8fad67cfbd3c3c22: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260923 20:08:58.915776 3971 raft_consensus.cc:348] T 906c58e7c4714f7bad7b4f568bfcebd7 P 98967392b7164818a63ed7e9912395c5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "659357c7635644ea8fad67cfbd3c3c22" member_type: VOTER last_known_addr { host: "127.1.170.131" port: 37945 } } peers { permanent_uuid: "98967392b7164818a63ed7e9912395c5" member_type: VOTER last_known_addr { host: "127.1.170.129" port: 44623 } } peers { permanent_uuid: "df7bc6713ea546638a209a6b211dfa0c" member_type: VOTER last_known_addr { host: "127.1.170.130" port: 39109 } }
I20260923 20:08:58.916193 3971 raft_consensus.cc:374] T 906c58e7c4714f7bad7b4f568bfcebd7 P 98967392b7164818a63ed7e9912395c5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:08:58.916298 3971 raft_consensus.cc:729] T 906c58e7c4714f7bad7b4f568bfcebd7 P 98967392b7164818a63ed7e9912395c5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98967392b7164818a63ed7e9912395c5, State: Initialized, Role: FOLLOWER
I20260923 20:08:58.917008 3971 consensus_queue.cc:260] T 906c58e7c4714f7bad7b4f568bfcebd7 P 98967392b7164818a63ed7e9912395c5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "659357c7635644ea8fad67cfbd3c3c22" member_type: VOTER last_known_addr { host: "127.1.170.131" port: 37945 } } peers { permanent_uuid: "98967392b7164818a63ed7e9912395c5" member_type: VOTER last_known_addr { host: "127.1.170.129" port: 44623 } } peers { permanent_uuid: "df7bc6713ea546638a209a6b211dfa0c" member_type: VOTER last_known_addr { host: "127.1.170.130" port: 39109 } }
I20260923 20:08:58.918931 3971 ts_tablet_manager.cc:1428] T 906c58e7c4714f7bad7b4f568bfcebd7 P 98967392b7164818a63ed7e9912395c5: Time spent starting tablet: real 0.012s user 0.010s sys 0.001s
I20260923 20:08:58.921548 3973 raft_consensus.cc:348] T 906c58e7c4714f7bad7b4f568bfcebd7 P df7bc6713ea546638a209a6b211dfa0c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "659357c7635644ea8fad67cfbd3c3c22" member_type: VOTER last_known_addr { host: "127.1.170.131" port: 37945 } } peers { permanent_uuid: "98967392b7164818a63ed7e9912395c5" member_type: VOTER last_known_addr { host: "127.1.170.129" port: 44623 } } peers { permanent_uuid: "df7bc6713ea546638a209a6b211dfa0c" member_type: VOTER last_known_addr { host: "127.1.170.130" port: 39109 } }
I20260923 20:08:58.922072 3973 raft_consensus.cc:374] T 906c58e7c4714f7bad7b4f568bfcebd7 P df7bc6713ea546638a209a6b211dfa0c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:08:58.922190 3973 raft_consensus.cc:729] T 906c58e7c4714f7bad7b4f568bfcebd7 P df7bc6713ea546638a209a6b211dfa0c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df7bc6713ea546638a209a6b211dfa0c, State: Initialized, Role: FOLLOWER
I20260923 20:08:58.922220 3972 raft_consensus.cc:348] T 906c58e7c4714f7bad7b4f568bfcebd7 P 659357c7635644ea8fad67cfbd3c3c22 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "659357c7635644ea8fad67cfbd3c3c22" member_type: VOTER last_known_addr { host: "127.1.170.131" port: 37945 } } peers { permanent_uuid: "98967392b7164818a63ed7e9912395c5" member_type: VOTER last_known_addr { host: "127.1.170.129" port: 44623 } } peers { permanent_uuid: "df7bc6713ea546638a209a6b211dfa0c" member_type: VOTER last_known_addr { host: "127.1.170.130" port: 39109 } }
I20260923 20:08:58.922621 3972 raft_consensus.cc:374] T 906c58e7c4714f7bad7b4f568bfcebd7 P 659357c7635644ea8fad67cfbd3c3c22 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:08:58.922729 3972 raft_consensus.cc:729] T 906c58e7c4714f7bad7b4f568bfcebd7 P 659357c7635644ea8fad67cfbd3c3c22 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 659357c7635644ea8fad67cfbd3c3c22, State: Initialized, Role: FOLLOWER
I20260923 20:08:58.922859 3973 consensus_queue.cc:260] T 906c58e7c4714f7bad7b4f568bfcebd7 P df7bc6713ea546638a209a6b211dfa0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "659357c7635644ea8fad67cfbd3c3c22" member_type: VOTER last_known_addr { host: "127.1.170.131" port: 37945 } } peers { permanent_uuid: "98967392b7164818a63ed7e9912395c5" member_type: VOTER last_known_addr { host: "127.1.170.129" port: 44623 } } peers { permanent_uuid: "df7bc6713ea546638a209a6b211dfa0c" member_type: VOTER last_known_addr { host: "127.1.170.130" port: 39109 } }
I20260923 20:08:58.923401 3972 consensus_queue.cc:260] T 906c58e7c4714f7bad7b4f568bfcebd7 P 659357c7635644ea8fad67cfbd3c3c22 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "659357c7635644ea8fad67cfbd3c3c22" member_type: VOTER last_known_addr { host: "127.1.170.131" port: 37945 } } peers { permanent_uuid: "98967392b7164818a63ed7e9912395c5" member_type: VOTER last_known_addr { host: "127.1.170.129" port: 44623 } } peers { permanent_uuid: "df7bc6713ea546638a209a6b211dfa0c" member_type: VOTER last_known_addr { host: "127.1.170.130" port: 39109 } }
I20260923 20:08:58.924031 3812 heartbeater.cc:503] Master 127.1.170.190:35033 was elected leader, sending a full tablet report...
I20260923 20:08:58.924888 3973 ts_tablet_manager.cc:1428] T 906c58e7c4714f7bad7b4f568bfcebd7 P df7bc6713ea546638a209a6b211dfa0c: Time spent starting tablet: real 0.015s user 0.012s sys 0.001s
I20260923 20:08:58.925537 3952 heartbeater.cc:503] Master 127.1.170.190:35033 was elected leader, sending a full tablet report...
I20260923 20:08:58.926095 3972 ts_tablet_manager.cc:1428] T 906c58e7c4714f7bad7b4f568bfcebd7 P 659357c7635644ea8fad67cfbd3c3c22: Time spent starting tablet: real 0.015s user 0.009s sys 0.005s
I20260923 20:08:58.993199 3979 raft_consensus.cc:482] T 906c58e7c4714f7bad7b4f568bfcebd7 P 659357c7635644ea8fad67cfbd3c3c22 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 20:08:58.993573 3979 raft_consensus.cc:504] T 906c58e7c4714f7bad7b4f568bfcebd7 P 659357c7635644ea8fad67cfbd3c3c22 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "659357c7635644ea8fad67cfbd3c3c22" member_type: VOTER last_known_addr { host: "127.1.170.131" port: 37945 } } peers { permanent_uuid: "98967392b7164818a63ed7e9912395c5" member_type: VOTER last_known_addr { host: "127.1.170.129" port: 44623 } } peers { permanent_uuid: "df7bc6713ea546638a209a6b211dfa0c" member_type: VOTER last_known_addr { host: "127.1.170.130" port: 39109 } }
I20260923 20:08:58.995713 3979 leader_election.cc:288] T 906c58e7c4714f7bad7b4f568bfcebd7 P 659357c7635644ea8fad67cfbd3c3c22 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 98967392b7164818a63ed7e9912395c5 (127.1.170.129:44623), df7bc6713ea546638a209a6b211dfa0c (127.1.170.130:39109)
W20260923 20:08:59.000914 3953 tablet.cc:2369] T 906c58e7c4714f7bad7b4f568bfcebd7 P 659357c7635644ea8fad67cfbd3c3c22: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 20:08:59.001819 3968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:08:58.996012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.170.131:55389 (local address 127.1.170.129:44623)
0923 20:08:58.996268 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0923 20:08:58.996273 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 20:08:58.996355 (+ 82us) server_negotiation.cc:407] Connection header received
0923 20:08:58.996554 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:08:58.996561 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:08:58.996675 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:08:58.996767 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:08:58.997544 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:08:58.998404 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:08:59.000006 (+ 1602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:08:59.000882 (+ 876us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:08:59.001056 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:08:59.001156 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0923 20:08:59.001429 (+ 273us) server_negotiation.cc:299] Negotiation successful
0923 20:08:59.001560 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260923 20:08:59.001833 3841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:08:58.996048 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.129:44623 (local address 127.1.170.131:55389)
0923 20:08:58.996251 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0923 20:08:58.996272 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0923 20:08:58.996408 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:08:58.997103 (+ 695us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:08:58.997111 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:08:58.997137 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:08:58.997399 (+ 262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:08:58.997410 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:08:58.998525 (+ 1115us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:08:58.998533 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:08:58.999840 (+ 1307us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:08:58.999851 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:08:59.001047 (+ 1196us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:08:59.001053 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:08:59.001249 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:08:59.001288 (+ 39us) client_negotiation.cc:769] Sending connection context
0923 20:08:59.001381 (+ 93us) client_negotiation.cc:240] Negotiation successful
0923 20:08:59.001528 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":60}
I20260923 20:08:59.002914 3970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:08:58.996758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.170.131:49237 (local address 127.1.170.130:39109)
0923 20:08:58.997022 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0923 20:08:58.997028 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:08:58.997461 (+ 433us) server_negotiation.cc:407] Connection header received
0923 20:08:58.997647 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:08:58.997653 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:08:58.997719 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:08:58.997814 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:08:58.998384 (+ 570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:08:58.999073 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:08:59.000744 (+ 1671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:08:59.001899 (+ 1155us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:08:59.002066 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:08:59.002167 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0923 20:08:59.002530 (+ 363us) server_negotiation.cc:299] Negotiation successful
0923 20:08:59.002663 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124}
I20260923 20:08:59.002954 3980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:08:58.996464 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.130:39109 (local address 127.1.170.131:49237)
0923 20:08:58.997361 (+ 897us) negotiation.cc:107] Waiting for socket to connect
0923 20:08:58.997376 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0923 20:08:58.997506 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:08:58.997837 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:08:58.997841 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:08:58.997863 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:08:58.998234 (+ 371us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:08:58.998245 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:08:58.999224 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:08:58.999230 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:08:59.000578 (+ 1348us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:08:59.000588 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:08:59.002253 (+ 1665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:08:59.002258 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:08:59.002373 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:08:59.002395 (+ 22us) client_negotiation.cc:769] Sending connection context
0923 20:08:59.002505 (+ 110us) client_negotiation.cc:240] Negotiation successful
0923 20:08:59.002638 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":716,"thread_start_us":526,"threads_started":1}
I20260923 20:08:59.002882 3628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "906c58e7c4714f7bad7b4f568bfcebd7" candidate_uuid: "659357c7635644ea8fad67cfbd3c3c22" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98967392b7164818a63ed7e9912395c5" is_pre_election: true
I20260923 20:08:59.003470 3628 raft_consensus.cc:2454] T 906c58e7c4714f7bad7b4f568bfcebd7 P 98967392b7164818a63ed7e9912395c5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 659357c7635644ea8fad67cfbd3c3c22 in term 0.
I20260923 20:08:59.004411 3833 leader_election.cc:302] T 906c58e7c4714f7bad7b4f568bfcebd7 P 659357c7635644ea8fad67cfbd3c3c22 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 659357c7635644ea8fad67cfbd3c3c22, 98967392b7164818a63ed7e9912395c5; no voters:
I20260923 20:08:59.004590 3762 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "906c58e7c4714f7bad7b4f568bfcebd7" candidate_uuid: "659357c7635644ea8fad67cfbd3c3c22" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df7bc6713ea546638a209a6b211dfa0c" is_pre_election: true
I20260923 20:08:59.005038 3979 raft_consensus.cc:2781] T 906c58e7c4714f7bad7b4f568bfcebd7 P 659357c7635644ea8fad67cfbd3c3c22 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 20:08:59.005177 3979 raft_consensus.cc:482] T 906c58e7c4714f7bad7b4f568bfcebd7 P 659357c7635644ea8fad67cfbd3c3c22 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 20:08:59.005245 3979 raft_consensus.cc:3037] T 906c58e7c4714f7bad7b4f568bfcebd7 P 659357c7635644ea8fad67cfbd3c3c22 [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:08:59.005255 3762 raft_consensus.cc:2454] T 906c58e7c4714f7bad7b4f568bfcebd7 P df7bc6713ea546638a209a6b211dfa0c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 659357c7635644ea8fad67cfbd3c3c22 in term 0.
I20260923 20:08:59.007616 3979 raft_consensus.cc:504] T 906c58e7c4714f7bad7b4f568bfcebd7 P 659357c7635644ea8fad67cfbd3c3c22 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "659357c7635644ea8fad67cfbd3c3c22" member_type: VOTER last_known_addr { host: "127.1.170.131" port: 37945 } } peers { permanent_uuid: "98967392b7164818a63ed7e9912395c5" member_type: VOTER last_known_addr { host: "127.1.170.129" port: 44623 } } peers { permanent_uuid: "df7bc6713ea546638a209a6b211dfa0c" member_type: VOTER last_known_addr { host: "127.1.170.130" port: 39109 } }
I20260923 20:08:59.008229 3979 leader_election.cc:288] T 906c58e7c4714f7bad7b4f568bfcebd7 P 659357c7635644ea8fad67cfbd3c3c22 [CANDIDATE]: Term 1 election: Requested vote from peers 98967392b7164818a63ed7e9912395c5 (127.1.170.129:44623), df7bc6713ea546638a209a6b211dfa0c (127.1.170.130:39109)
I20260923 20:08:59.008991 3628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "906c58e7c4714f7bad7b4f568bfcebd7" candidate_uuid: "659357c7635644ea8fad67cfbd3c3c22" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98967392b7164818a63ed7e9912395c5"
I20260923 20:08:59.008991 3762 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "906c58e7c4714f7bad7b4f568bfcebd7" candidate_uuid: "659357c7635644ea8fad67cfbd3c3c22" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df7bc6713ea546638a209a6b211dfa0c"
I20260923 20:08:59.009235 3628 raft_consensus.cc:3037] T 906c58e7c4714f7bad7b4f568bfcebd7 P 98967392b7164818a63ed7e9912395c5 [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:08:59.009230 3762 raft_consensus.cc:3037] T 906c58e7c4714f7bad7b4f568bfcebd7 P df7bc6713ea546638a209a6b211dfa0c [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:08:59.013319 3762 raft_consensus.cc:2454] T 906c58e7c4714f7bad7b4f568bfcebd7 P df7bc6713ea546638a209a6b211dfa0c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 659357c7635644ea8fad67cfbd3c3c22 in term 1.
I20260923 20:08:59.013296 3628 raft_consensus.cc:2454] T 906c58e7c4714f7bad7b4f568bfcebd7 P 98967392b7164818a63ed7e9912395c5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 659357c7635644ea8fad67cfbd3c3c22 in term 1.
I20260923 20:08:59.014065 3833 leader_election.cc:302] T 906c58e7c4714f7bad7b4f568bfcebd7 P 659357c7635644ea8fad67cfbd3c3c22 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 659357c7635644ea8fad67cfbd3c3c22, df7bc6713ea546638a209a6b211dfa0c; no voters:
I20260923 20:08:59.014509 3979 raft_consensus.cc:2781] T 906c58e7c4714f7bad7b4f568bfcebd7 P 659357c7635644ea8fad67cfbd3c3c22 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 20:08:59.015242 3979 raft_consensus.cc:686] T 906c58e7c4714f7bad7b4f568bfcebd7 P 659357c7635644ea8fad67cfbd3c3c22 [term 1 LEADER]: Becoming Leader. State: Replica: 659357c7635644ea8fad67cfbd3c3c22, State: Running, Role: LEADER
I20260923 20:08:59.015874 3979 consensus_queue.cc:237] T 906c58e7c4714f7bad7b4f568bfcebd7 P 659357c7635644ea8fad67cfbd3c3c22 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "659357c7635644ea8fad67cfbd3c3c22" member_type: VOTER last_known_addr { host: "127.1.170.131" port: 37945 } } peers { permanent_uuid: "98967392b7164818a63ed7e9912395c5" member_type: VOTER last_known_addr { host: "127.1.170.129" port: 44623 } } peers { permanent_uuid: "df7bc6713ea546638a209a6b211dfa0c" member_type: VOTER last_known_addr { host: "127.1.170.130" port: 39109 } }
I20260923 20:08:59.023203 3466 catalog_manager.cc:5495] T 906c58e7c4714f7bad7b4f568bfcebd7 P 659357c7635644ea8fad67cfbd3c3c22 reported cstate change: term changed from 0 to 1, leader changed from <none> to 659357c7635644ea8fad67cfbd3c3c22 (127.1.170.131). New cstate: current_term: 1 leader_uuid: "659357c7635644ea8fad67cfbd3c3c22" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "659357c7635644ea8fad67cfbd3c3c22" member_type: VOTER last_known_addr { host: "127.1.170.131" port: 37945 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "98967392b7164818a63ed7e9912395c5" member_type: VOTER last_known_addr { host: "127.1.170.129" port: 44623 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df7bc6713ea546638a209a6b211dfa0c" member_type: VOTER last_known_addr { host: "127.1.170.130" port: 39109 } health_report { overall_health: UNKNOWN } } }
Sep 23 20:08:59 dist-test-slave-th4g krb5kdc[3421](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194136, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.170.190@KRBTEST.COM for kudu/127.1.170.190@KRBTEST.COM
I20260923 20:08:59.114890 3988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:08:59.097661 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.190:35033 (local address 127.0.0.1:34718)
0923 20:08:59.098375 (+ 714us) negotiation.cc:107] Waiting for socket to connect
0923 20:08:59.098397 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0923 20:08:59.098542 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:08:59.098947 (+ 405us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:08:59.098952 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:08:59.100106 (+ 1154us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:08:59.100395 (+ 289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:08:59.100407 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:08:59.101756 (+ 1349us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:08:59.101761 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:08:59.102909 (+ 1148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:08:59.102922 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:08:59.103765 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:08:59.103770 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:08:59.103890 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:08:59.104582 (+ 692us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:08:59.104599 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:08:59.107469 (+ 2870us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:08:59.111873 (+ 4404us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:08:59.111892 (+ 19us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:08:59.111898 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:08:59.112435 (+ 537us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:08:59.112981 (+ 546us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:08:59.112985 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:08:59.112988 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:08:59.113233 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:08:59.113846 (+ 613us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:08:59.113863 (+ 17us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:08:59.114058 (+ 195us) client_negotiation.cc:769] Sending connection context
0923 20:08:59.114228 (+ 170us) client_negotiation.cc:240] Negotiation successful
0923 20:08:59.114491 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":577,"thread_start_us":426,"threads_started":1}
I20260923 20:08:59.114998 3840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:08:59.097425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34718 (local address 127.1.170.190:35033)
0923 20:08:59.097791 (+ 366us) server_negotiation.cc:206] Beginning negotiation
0923 20:08:59.097798 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:08:59.098524 (+ 726us) server_negotiation.cc:407] Connection header received
0923 20:08:59.098650 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:08:59.098655 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:08:59.098715 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:08:59.098806 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:08:59.100586 (+ 1780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:08:59.101568 (+ 982us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:08:59.103076 (+ 1508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:08:59.103559 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:08:59.103665 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:08:59.107779 (+ 4114us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:08:59.107785 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:08:59.107789 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:08:59.107825 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:08:59.111642 (+ 3817us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:08:59.112689 (+ 1047us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:08:59.112694 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:08:59.112697 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:08:59.112841 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:08:59.113390 (+ 549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:08:59.113394 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:08:59.113396 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:08:59.113693 (+ 297us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:08:59.113816 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0923 20:08:59.114366 (+ 550us) server_negotiation.cc:299] Negotiation successful
0923 20:08:59.114662 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":100,"server-negotiator.queue_time_us":67}
I20260923 20:08:59.121400 3466 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.1.170.190@KRBTEST.COM'} at 127.0.0.1:34718:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260923 20:08:59.122740 3466 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 20:08:59.132516 3603 tablet_service.cc:1467] Processing CreateTablet for tablet 27fa16f0cfb047fd8779810f3c13977e (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=2b3dd5ebac7748c7818aaa7ef1cd6448]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260923 20:08:59.132555 3742 tablet_service.cc:1467] Processing CreateTablet for tablet 27fa16f0cfb047fd8779810f3c13977e (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=2b3dd5ebac7748c7818aaa7ef1cd6448]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260923 20:08:59.132654 3882 tablet_service.cc:1467] Processing CreateTablet for tablet 27fa16f0cfb047fd8779810f3c13977e (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=2b3dd5ebac7748c7818aaa7ef1cd6448]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260923 20:08:59.133500 3603 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 27fa16f0cfb047fd8779810f3c13977e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:08:59.133505 3882 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 27fa16f0cfb047fd8779810f3c13977e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:08:59.133505 3742 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 27fa16f0cfb047fd8779810f3c13977e. 1 dirs total, 0 dirs full, 0 dirs failed
W20260923 20:08:59.138616 3674 tablet.cc:2369] T 906c58e7c4714f7bad7b4f568bfcebd7 P 98967392b7164818a63ed7e9912395c5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 20:08:59.142778 3972 tablet_bootstrap.cc:492] T 27fa16f0cfb047fd8779810f3c13977e P 659357c7635644ea8fad67cfbd3c3c22: Bootstrap starting.
I20260923 20:08:59.143014 3971 tablet_bootstrap.cc:492] T 27fa16f0cfb047fd8779810f3c13977e P 98967392b7164818a63ed7e9912395c5: Bootstrap starting.
I20260923 20:08:59.143985 3973 tablet_bootstrap.cc:492] T 27fa16f0cfb047fd8779810f3c13977e P df7bc6713ea546638a209a6b211dfa0c: Bootstrap starting.
I20260923 20:08:59.145040 3971 tablet_bootstrap.cc:654] T 27fa16f0cfb047fd8779810f3c13977e P 98967392b7164818a63ed7e9912395c5: Neither blocks nor log segments found. Creating new log.
I20260923 20:08:59.145457 3972 tablet_bootstrap.cc:654] T 27fa16f0cfb047fd8779810f3c13977e P 659357c7635644ea8fad67cfbd3c3c22: Neither blocks nor log segments found. Creating new log.
I20260923 20:08:59.146279 3973 tablet_bootstrap.cc:654] T 27fa16f0cfb047fd8779810f3c13977e P df7bc6713ea546638a209a6b211dfa0c: Neither blocks nor log segments found. Creating new log.
I20260923 20:08:59.148343 3971 tablet_bootstrap.cc:492] T 27fa16f0cfb047fd8779810f3c13977e P 98967392b7164818a63ed7e9912395c5: No bootstrap required, opened a new log
I20260923 20:08:59.148711 3971 ts_tablet_manager.cc:1397] T 27fa16f0cfb047fd8779810f3c13977e P 98967392b7164818a63ed7e9912395c5: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260923 20:08:59.150424 3971 raft_consensus.cc:348] T 27fa16f0cfb047fd8779810f3c13977e P 98967392b7164818a63ed7e9912395c5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98967392b7164818a63ed7e9912395c5" member_type: VOTER last_known_addr { host: "127.1.170.129" port: 44623 } } peers { permanent_uuid: "df7bc6713ea546638a209a6b211dfa0c" member_type: VOTER last_known_addr { host: "127.1.170.130" port: 39109 } } peers { permanent_uuid: "659357c7635644ea8fad67cfbd3c3c22" member_type: VOTER last_known_addr { host: "127.1.170.131" port: 37945 } }
I20260923 20:08:59.150818 3971 raft_consensus.cc:374] T 27fa16f0cfb047fd8779810f3c13977e P 98967392b7164818a63ed7e9912395c5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:08:59.150920 3971 raft_consensus.cc:729] T 27fa16f0cfb047fd8779810f3c13977e P 98967392b7164818a63ed7e9912395c5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98967392b7164818a63ed7e9912395c5, State: Initialized, Role: FOLLOWER
I20260923 20:08:59.151245 3971 consensus_queue.cc:260] T 27fa16f0cfb047fd8779810f3c13977e P 98967392b7164818a63ed7e9912395c5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98967392b7164818a63ed7e9912395c5" member_type: VOTER last_known_addr { host: "127.1.170.129" port: 44623 } } peers { permanent_uuid: "df7bc6713ea546638a209a6b211dfa0c" member_type: VOTER last_known_addr { host: "127.1.170.130" port: 39109 } } peers { permanent_uuid: "659357c7635644ea8fad67cfbd3c3c22" member_type: VOTER last_known_addr { host: "127.1.170.131" port: 37945 } }
I20260923 20:08:59.151757 3977 tablet_replica.cc:439] T 27fa16f0cfb047fd8779810f3c13977e P 98967392b7164818a63ed7e9912395c5: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98967392b7164818a63ed7e9912395c5" member_type: VOTER last_known_addr { host: "127.1.170.129" port: 44623 } } peers { permanent_uuid: "df7bc6713ea546638a209a6b211dfa0c" member_type: VOTER last_known_addr { host: "127.1.170.130" port: 39109 } } peers { permanent_uuid: "659357c7635644ea8fad67cfbd3c3c22" member_type: VOTER last_known_addr { host: "127.1.170.131" port: 37945 } } }
I20260923 20:08:59.152153 3977 tablet_replica.cc:442] T 27fa16f0cfb047fd8779810f3c13977e P 98967392b7164818a63ed7e9912395c5: This TxnStatusTablet replica's current role is: FOLLOWER
I20260923 20:08:59.152307 3972 tablet_bootstrap.cc:492] T 27fa16f0cfb047fd8779810f3c13977e P 659357c7635644ea8fad67cfbd3c3c22: No bootstrap required, opened a new log
I20260923 20:08:59.152484 3972 ts_tablet_manager.cc:1397] T 27fa16f0cfb047fd8779810f3c13977e P 659357c7635644ea8fad67cfbd3c3c22: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.002s
I20260923 20:08:59.152770 3973 tablet_bootstrap.cc:492] T 27fa16f0cfb047fd8779810f3c13977e P df7bc6713ea546638a209a6b211dfa0c: No bootstrap required, opened a new log
I20260923 20:08:59.152807 3971 ts_tablet_manager.cc:1428] T 27fa16f0cfb047fd8779810f3c13977e P 98967392b7164818a63ed7e9912395c5: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260923 20:08:59.153014 3973 ts_tablet_manager.cc:1397] T 27fa16f0cfb047fd8779810f3c13977e P df7bc6713ea546638a209a6b211dfa0c: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260923 20:08:59.153366 3972 raft_consensus.cc:348] T 27fa16f0cfb047fd8779810f3c13977e P 659357c7635644ea8fad67cfbd3c3c22 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98967392b7164818a63ed7e9912395c5" member_type: VOTER last_known_addr { host: "127.1.170.129" port: 44623 } } peers { permanent_uuid: "df7bc6713ea546638a209a6b211dfa0c" member_type: VOTER last_known_addr { host: "127.1.170.130" port: 39109 } } peers { permanent_uuid: "659357c7635644ea8fad67cfbd3c3c22" member_type: VOTER last_known_addr { host: "127.1.170.131" port: 37945 } }
I20260923 20:08:59.153596 3972 raft_consensus.cc:374] T 27fa16f0cfb047fd8779810f3c13977e P 659357c7635644ea8fad67cfbd3c3c22 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:08:59.153712 3972 raft_consensus.cc:729] T 27fa16f0cfb047fd8779810f3c13977e P 659357c7635644ea8fad67cfbd3c3c22 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 659357c7635644ea8fad67cfbd3c3c22, State: Initialized, Role: FOLLOWER
I20260923 20:08:59.153913 3973 raft_consensus.cc:348] T 27fa16f0cfb047fd8779810f3c13977e P df7bc6713ea546638a209a6b211dfa0c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98967392b7164818a63ed7e9912395c5" member_type: VOTER last_known_addr { host: "127.1.170.129" port: 44623 } } peers { permanent_uuid: "df7bc6713ea546638a209a6b211dfa0c" member_type: VOTER last_known_addr { host: "127.1.170.130" port: 39109 } } peers { permanent_uuid: "659357c7635644ea8fad67cfbd3c3c22" member_type: VOTER last_known_addr { host: "127.1.170.131" port: 37945 } }
I20260923 20:08:59.154044 3972 consensus_queue.cc:260] T 27fa16f0cfb047fd8779810f3c13977e P 659357c7635644ea8fad67cfbd3c3c22 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98967392b7164818a63ed7e9912395c5" member_type: VOTER last_known_addr { host: "127.1.170.129" port: 44623 } } peers { permanent_uuid: "df7bc6713ea546638a209a6b211dfa0c" member_type: VOTER last_known_addr { host: "127.1.170.130" port: 39109 } } peers { permanent_uuid: "659357c7635644ea8fad67cfbd3c3c22" member_type: VOTER last_known_addr { host: "127.1.170.131" port: 37945 } }
I20260923 20:08:59.154176 3973 raft_consensus.cc:374] T 27fa16f0cfb047fd8779810f3c13977e P df7bc6713ea546638a209a6b211dfa0c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:08:59.154291 3973 raft_consensus.cc:729] T 27fa16f0cfb047fd8779810f3c13977e P df7bc6713ea546638a209a6b211dfa0c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df7bc6713ea546638a209a6b211dfa0c, State: Initialized, Role: FOLLOWER
I20260923 20:08:59.154527 3973 consensus_queue.cc:260] T 27fa16f0cfb047fd8779810f3c13977e P df7bc6713ea546638a209a6b211dfa0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98967392b7164818a63ed7e9912395c5" member_type: VOTER last_known_addr { host: "127.1.170.129" port: 44623 } } peers { permanent_uuid: "df7bc6713ea546638a209a6b211dfa0c" member_type: VOTER last_known_addr { host: "127.1.170.130" port: 39109 } } peers { permanent_uuid: "659357c7635644ea8fad67cfbd3c3c22" member_type: VOTER last_known_addr { host: "127.1.170.131" port: 37945 } }
I20260923 20:08:59.154747 3972 ts_tablet_manager.cc:1428] T 27fa16f0cfb047fd8779810f3c13977e P 659357c7635644ea8fad67cfbd3c3c22: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260923 20:08:59.154702 3979 tablet_replica.cc:439] T 27fa16f0cfb047fd8779810f3c13977e P 659357c7635644ea8fad67cfbd3c3c22: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98967392b7164818a63ed7e9912395c5" member_type: VOTER last_known_addr { host: "127.1.170.129" port: 44623 } } peers { permanent_uuid: "df7bc6713ea546638a209a6b211dfa0c" member_type: VOTER last_known_addr { host: "127.1.170.130" port: 39109 } } peers { permanent_uuid: "659357c7635644ea8fad67cfbd3c3c22" member_type: VOTER last_known_addr { host: "127.1.170.131" port: 37945 } } }
I20260923 20:08:59.154994 3979 tablet_replica.cc:442] T 27fa16f0cfb047fd8779810f3c13977e P 659357c7635644ea8fad67cfbd3c3c22: This TxnStatusTablet replica's current role is: FOLLOWER
I20260923 20:08:59.155125 3973 ts_tablet_manager.cc:1428] T 27fa16f0cfb047fd8779810f3c13977e P df7bc6713ea546638a209a6b211dfa0c: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20260923 20:08:59.155387 3978 tablet_replica.cc:439] T 27fa16f0cfb047fd8779810f3c13977e P df7bc6713ea546638a209a6b211dfa0c: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98967392b7164818a63ed7e9912395c5" member_type: VOTER last_known_addr { host: "127.1.170.129" port: 44623 } } peers { permanent_uuid: "df7bc6713ea546638a209a6b211dfa0c" member_type: VOTER last_known_addr { host: "127.1.170.130" port: 39109 } } peers { permanent_uuid: "659357c7635644ea8fad67cfbd3c3c22" member_type: VOTER last_known_addr { host: "127.1.170.131" port: 37945 } } }
I20260923 20:08:59.155740 3978 tablet_replica.cc:442] T 27fa16f0cfb047fd8779810f3c13977e P df7bc6713ea546638a209a6b211dfa0c: This TxnStatusTablet replica's current role is: FOLLOWER
W20260923 20:08:59.169108 3813 tablet.cc:2369] T 27fa16f0cfb047fd8779810f3c13977e P df7bc6713ea546638a209a6b211dfa0c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 20:08:59.388728 3977 raft_consensus.cc:482] T 27fa16f0cfb047fd8779810f3c13977e P 98967392b7164818a63ed7e9912395c5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 20:08:59.388973 3977 raft_consensus.cc:504] T 27fa16f0cfb047fd8779810f3c13977e P 98967392b7164818a63ed7e9912395c5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98967392b7164818a63ed7e9912395c5" member_type: VOTER last_known_addr { host: "127.1.170.129" port: 44623 } } peers { permanent_uuid: "df7bc6713ea546638a209a6b211dfa0c" member_type: VOTER last_known_addr { host: "127.1.170.130" port: 39109 } } peers { permanent_uuid: "659357c7635644ea8fad67cfbd3c3c22" member_type: VOTER last_known_addr { host: "127.1.170.131" port: 37945 } }
I20260923 20:08:59.390868 3977 leader_election.cc:288] T 27fa16f0cfb047fd8779810f3c13977e P 98967392b7164818a63ed7e9912395c5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df7bc6713ea546638a209a6b211dfa0c (127.1.170.130:39109), 659357c7635644ea8fad67cfbd3c3c22 (127.1.170.131:37945)
I20260923 20:08:59.400393 3992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:08:59.391644 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.170.129:40219 (local address 127.1.170.131:37945)
0923 20:08:59.394169 (+ 2525us) server_negotiation.cc:206] Beginning negotiation
0923 20:08:59.394175 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:08:59.394683 (+ 508us) server_negotiation.cc:407] Connection header received
0923 20:08:59.394888 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:08:59.394893 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:08:59.394964 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:08:59.395072 (+ 108us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:08:59.395641 (+ 569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:08:59.396328 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:08:59.398599 (+ 2271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:08:59.399369 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:08:59.399578 (+ 209us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:08:59.399656 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0923 20:08:59.399922 (+ 266us) server_negotiation.cc:299] Negotiation successful
0923 20:08:59.400052 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2410,"thread_start_us":2323,"threads_started":1}
I20260923 20:08:59.400440 3993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:08:59.391583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.131:37945 (local address 127.1.170.129:40219)
0923 20:08:59.394569 (+ 2986us) negotiation.cc:107] Waiting for socket to connect
0923 20:08:59.394584 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0923 20:08:59.394753 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:08:59.395101 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:08:59.395105 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:08:59.395129 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:08:59.395488 (+ 359us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:08:59.395496 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:08:59.396498 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:08:59.396503 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:08:59.398440 (+ 1937us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:08:59.398454 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:08:59.399580 (+ 1126us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:08:59.399586 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:08:59.399757 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:08:59.399780 (+ 23us) client_negotiation.cc:769] Sending connection context
0923 20:08:59.399883 (+ 103us) client_negotiation.cc:240] Negotiation successful
0923 20:08:59.400020 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2874,"mutex_wait_us":25,"spinlock_wait_cycles":21376,"thread_start_us":2013,"threads_started":1}
I20260923 20:08:59.401364 3970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:08:59.391262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.170.129:46781 (local address 127.1.170.130:39109)
0923 20:08:59.391511 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0923 20:08:59.391518 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:08:59.394112 (+ 2594us) server_negotiation.cc:407] Connection header received
0923 20:08:59.394413 (+ 301us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:08:59.394419 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:08:59.394490 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:08:59.394591 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:08:59.395387 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:08:59.396337 (+ 950us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:08:59.399508 (+ 3171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:08:59.400344 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:08:59.400546 (+ 202us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:08:59.400634 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0923 20:08:59.400980 (+ 346us) server_negotiation.cc:299] Negotiation successful
0923 20:08:59.401097 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260923 20:08:59.401402 3994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:08:59.391569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.130:39109 (local address 127.1.170.129:46781)
0923 20:08:59.394004 (+ 2435us) negotiation.cc:107] Waiting for socket to connect
0923 20:08:59.394026 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0923 20:08:59.394248 (+ 222us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:08:59.394640 (+ 392us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:08:59.394652 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:08:59.394684 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:08:59.395080 (+ 396us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:08:59.395089 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:08:59.396752 (+ 1663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:08:59.396764 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:08:59.399304 (+ 2540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:08:59.399319 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:08:59.400589 (+ 1270us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:08:59.400595 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:08:59.400775 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:08:59.400833 (+ 58us) client_negotiation.cc:769] Sending connection context
0923 20:08:59.400941 (+ 108us) client_negotiation.cc:240] Negotiation successful
0923 20:08:59.401090 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2239,"thread_start_us":2641,"threads_started":1}
I20260923 20:08:59.401687 3902 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27fa16f0cfb047fd8779810f3c13977e" candidate_uuid: "98967392b7164818a63ed7e9912395c5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "659357c7635644ea8fad67cfbd3c3c22" is_pre_election: true
I20260923 20:08:59.402145 3902 raft_consensus.cc:2454] T 27fa16f0cfb047fd8779810f3c13977e P 659357c7635644ea8fad67cfbd3c3c22 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 98967392b7164818a63ed7e9912395c5 in term 0.
I20260923 20:08:59.402149 3762 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27fa16f0cfb047fd8779810f3c13977e" candidate_uuid: "98967392b7164818a63ed7e9912395c5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df7bc6713ea546638a209a6b211dfa0c" is_pre_election: true
I20260923 20:08:59.402408 3762 raft_consensus.cc:2454] T 27fa16f0cfb047fd8779810f3c13977e P df7bc6713ea546638a209a6b211dfa0c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 98967392b7164818a63ed7e9912395c5 in term 0.
I20260923 20:08:59.402885 3556 leader_election.cc:302] T 27fa16f0cfb047fd8779810f3c13977e P 98967392b7164818a63ed7e9912395c5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 659357c7635644ea8fad67cfbd3c3c22, 98967392b7164818a63ed7e9912395c5; no voters:
I20260923 20:08:59.403303 3977 raft_consensus.cc:2781] T 27fa16f0cfb047fd8779810f3c13977e P 98967392b7164818a63ed7e9912395c5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 20:08:59.403435 3977 raft_consensus.cc:482] T 27fa16f0cfb047fd8779810f3c13977e P 98967392b7164818a63ed7e9912395c5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 20:08:59.403540 3977 raft_consensus.cc:3037] T 27fa16f0cfb047fd8779810f3c13977e P 98967392b7164818a63ed7e9912395c5 [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:08:59.405997 3977 raft_consensus.cc:504] T 27fa16f0cfb047fd8779810f3c13977e P 98967392b7164818a63ed7e9912395c5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98967392b7164818a63ed7e9912395c5" member_type: VOTER last_known_addr { host: "127.1.170.129" port: 44623 } } peers { permanent_uuid: "df7bc6713ea546638a209a6b211dfa0c" member_type: VOTER last_known_addr { host: "127.1.170.130" port: 39109 } } peers { permanent_uuid: "659357c7635644ea8fad67cfbd3c3c22" member_type: VOTER last_known_addr { host: "127.1.170.131" port: 37945 } }
I20260923 20:08:59.406580 3977 leader_election.cc:288] T 27fa16f0cfb047fd8779810f3c13977e P 98967392b7164818a63ed7e9912395c5 [CANDIDATE]: Term 1 election: Requested vote from peers df7bc6713ea546638a209a6b211dfa0c (127.1.170.130:39109), 659357c7635644ea8fad67cfbd3c3c22 (127.1.170.131:37945)
I20260923 20:08:59.407096 3902 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27fa16f0cfb047fd8779810f3c13977e" candidate_uuid: "98967392b7164818a63ed7e9912395c5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "659357c7635644ea8fad67cfbd3c3c22"
I20260923 20:08:59.407289 3902 raft_consensus.cc:3037] T 27fa16f0cfb047fd8779810f3c13977e P 659357c7635644ea8fad67cfbd3c3c22 [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:08:59.407388 3762 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27fa16f0cfb047fd8779810f3c13977e" candidate_uuid: "98967392b7164818a63ed7e9912395c5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df7bc6713ea546638a209a6b211dfa0c"
I20260923 20:08:59.407783 3762 raft_consensus.cc:3037] T 27fa16f0cfb047fd8779810f3c13977e P df7bc6713ea546638a209a6b211dfa0c [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:08:59.409658 3902 raft_consensus.cc:2454] T 27fa16f0cfb047fd8779810f3c13977e P 659357c7635644ea8fad67cfbd3c3c22 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 98967392b7164818a63ed7e9912395c5 in term 1.
I20260923 20:08:59.410058 3762 raft_consensus.cc:2454] T 27fa16f0cfb047fd8779810f3c13977e P df7bc6713ea546638a209a6b211dfa0c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 98967392b7164818a63ed7e9912395c5 in term 1.
I20260923 20:08:59.410131 3556 leader_election.cc:302] T 27fa16f0cfb047fd8779810f3c13977e P 98967392b7164818a63ed7e9912395c5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 659357c7635644ea8fad67cfbd3c3c22, 98967392b7164818a63ed7e9912395c5; no voters:
I20260923 20:08:59.410499 3977 raft_consensus.cc:2781] T 27fa16f0cfb047fd8779810f3c13977e P 98967392b7164818a63ed7e9912395c5 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 20:08:59.411016 3977 raft_consensus.cc:686] T 27fa16f0cfb047fd8779810f3c13977e P 98967392b7164818a63ed7e9912395c5 [term 1 LEADER]: Becoming Leader. State: Replica: 98967392b7164818a63ed7e9912395c5, State: Running, Role: LEADER
I20260923 20:08:59.411692 3977 consensus_queue.cc:237] T 27fa16f0cfb047fd8779810f3c13977e P 98967392b7164818a63ed7e9912395c5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98967392b7164818a63ed7e9912395c5" member_type: VOTER last_known_addr { host: "127.1.170.129" port: 44623 } } peers { permanent_uuid: "df7bc6713ea546638a209a6b211dfa0c" member_type: VOTER last_known_addr { host: "127.1.170.130" port: 39109 } } peers { permanent_uuid: "659357c7635644ea8fad67cfbd3c3c22" member_type: VOTER last_known_addr { host: "127.1.170.131" port: 37945 } }
I20260923 20:08:59.415110 3995 tablet_replica.cc:439] T 27fa16f0cfb047fd8779810f3c13977e P 98967392b7164818a63ed7e9912395c5: TxnStatusTablet state changed. Reason: New leader 98967392b7164818a63ed7e9912395c5. Latest consensus state: current_term: 1 leader_uuid: "98967392b7164818a63ed7e9912395c5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98967392b7164818a63ed7e9912395c5" member_type: VOTER last_known_addr { host: "127.1.170.129" port: 44623 } } peers { permanent_uuid: "df7bc6713ea546638a209a6b211dfa0c" member_type: VOTER last_known_addr { host: "127.1.170.130" port: 39109 } } peers { permanent_uuid: "659357c7635644ea8fad67cfbd3c3c22" member_type: VOTER last_known_addr { host: "127.1.170.131" port: 37945 } } }
I20260923 20:08:59.415478 3995 tablet_replica.cc:442] T 27fa16f0cfb047fd8779810f3c13977e P 98967392b7164818a63ed7e9912395c5: This TxnStatusTablet replica's current role is: LEADER
I20260923 20:08:59.416167 3997 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260923 20:08:59.416452 3997 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260923 20:08:59.417132 3465 catalog_manager.cc:5495] T 27fa16f0cfb047fd8779810f3c13977e P 98967392b7164818a63ed7e9912395c5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 98967392b7164818a63ed7e9912395c5 (127.1.170.129). New cstate: current_term: 1 leader_uuid: "98967392b7164818a63ed7e9912395c5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98967392b7164818a63ed7e9912395c5" member_type: VOTER last_known_addr { host: "127.1.170.129" port: 44623 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "df7bc6713ea546638a209a6b211dfa0c" member_type: VOTER last_known_addr { host: "127.1.170.130" port: 39109 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "659357c7635644ea8fad67cfbd3c3c22" member_type: VOTER last_known_addr { host: "127.1.170.131" port: 37945 } health_report { overall_health: UNKNOWN } } }
I20260923 20:08:59.468128 3968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:08:59.460573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38932 (local address 127.1.170.129:44623)
0923 20:08:59.460738 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0923 20:08:59.460785 (+ 47us) server_negotiation.cc:399] Waiting for connection header
0923 20:08:59.460807 (+ 22us) server_negotiation.cc:407] Connection header received
0923 20:08:59.461287 (+ 480us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:08:59.461295 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:08:59.461422 (+ 127us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:08:59.461753 (+ 331us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 20:08:59.462267 (+ 514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:08:59.463529 (+ 1262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:08:59.465054 (+ 1525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:08:59.465520 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:08:59.465739 (+ 219us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:08:59.466348 (+ 609us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 20:08:59.467234 (+ 886us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 20:08:59.467416 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0923 20:08:59.467553 (+ 137us) server_negotiation.cc:299] Negotiation successful
0923 20:08:59.467710 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260923 20:08:59.468149 3988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:08:59.460460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.129:44623 (local address 127.0.0.1:38932)
0923 20:08:59.460701 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0923 20:08:59.460722 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0923 20:08:59.460888 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:08:59.461784 (+ 896us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:08:59.461791 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:08:59.461808 (+ 17us) client_negotiation.cc:189] Negotiated authn=TOKEN
0923 20:08:59.462104 (+ 296us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:08:59.462116 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:08:59.463723 (+ 1607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:08:59.463733 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:08:59.464865 (+ 1132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:08:59.464879 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:08:59.465752 (+ 873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:08:59.465759 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:08:59.466013 (+ 254us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:08:59.466107 (+ 94us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0923 20:08:59.467368 (+ 1261us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0923 20:08:59.467397 (+ 29us) client_negotiation.cc:769] Sending connection context
0923 20:08:59.467529 (+ 132us) client_negotiation.cc:240] Negotiation successful
0923 20:08:59.467789 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42}
I20260923 20:08:59.474902 3979 consensus_queue.cc:1035] T 906c58e7c4714f7bad7b4f568bfcebd7 P 659357c7635644ea8fad67cfbd3c3c22 [LEADER]: Connected to new peer: Peer: permanent_uuid: "98967392b7164818a63ed7e9912395c5" member_type: VOTER last_known_addr { host: "127.1.170.129" port: 44623 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 20:08:59.477506 3902 raft_consensus.cc:1261] T 27fa16f0cfb047fd8779810f3c13977e P 659357c7635644ea8fad67cfbd3c3c22 [term 1 FOLLOWER]: Refusing update from remote peer 98967392b7164818a63ed7e9912395c5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 20:08:59.478008 3762 raft_consensus.cc:1261] T 27fa16f0cfb047fd8779810f3c13977e P df7bc6713ea546638a209a6b211dfa0c [term 1 FOLLOWER]: Refusing update from remote peer 98967392b7164818a63ed7e9912395c5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 20:08:59.478392 3995 consensus_queue.cc:1035] T 27fa16f0cfb047fd8779810f3c13977e P 98967392b7164818a63ed7e9912395c5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "659357c7635644ea8fad67cfbd3c3c22" member_type: VOTER last_known_addr { host: "127.1.170.131" port: 37945 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 20:08:59.478835 3977 consensus_queue.cc:1035] T 27fa16f0cfb047fd8779810f3c13977e P 98967392b7164818a63ed7e9912395c5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df7bc6713ea546638a209a6b211dfa0c" member_type: VOTER last_known_addr { host: "127.1.170.130" port: 39109 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 20:08:59.501744 3978 tablet_replica.cc:439] T 27fa16f0cfb047fd8779810f3c13977e P df7bc6713ea546638a209a6b211dfa0c: TxnStatusTablet state changed. Reason: New leader 98967392b7164818a63ed7e9912395c5. Latest consensus state: current_term: 1 leader_uuid: "98967392b7164818a63ed7e9912395c5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98967392b7164818a63ed7e9912395c5" member_type: VOTER last_known_addr { host: "127.1.170.129" port: 44623 } } peers { permanent_uuid: "df7bc6713ea546638a209a6b211dfa0c" member_type: VOTER last_known_addr { host: "127.1.170.130" port: 39109 } } peers { permanent_uuid: "659357c7635644ea8fad67cfbd3c3c22" member_type: VOTER last_known_addr { host: "127.1.170.131" port: 37945 } } }
I20260923 20:08:59.502223 3978 tablet_replica.cc:442] T 27fa16f0cfb047fd8779810f3c13977e P df7bc6713ea546638a209a6b211dfa0c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260923 20:08:59.503237 3979 tablet_replica.cc:439] T 27fa16f0cfb047fd8779810f3c13977e P 659357c7635644ea8fad67cfbd3c3c22: TxnStatusTablet state changed. Reason: New leader 98967392b7164818a63ed7e9912395c5. Latest consensus state: current_term: 1 leader_uuid: "98967392b7164818a63ed7e9912395c5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98967392b7164818a63ed7e9912395c5" member_type: VOTER last_known_addr { host: "127.1.170.129" port: 44623 } } peers { permanent_uuid: "df7bc6713ea546638a209a6b211dfa0c" member_type: VOTER last_known_addr { host: "127.1.170.130" port: 39109 } } peers { permanent_uuid: "659357c7635644ea8fad67cfbd3c3c22" member_type: VOTER last_known_addr { host: "127.1.170.131" port: 37945 } } }
I20260923 20:08:59.503521 3979 tablet_replica.cc:442] T 27fa16f0cfb047fd8779810f3c13977e P 659357c7635644ea8fad67cfbd3c3c22: This TxnStatusTablet replica's current role is: FOLLOWER
I20260923 20:08:59.506269 4000 mvcc.cc:205] Tried to move back new op lower bound from 7332635195291435008 to 7332635195034107904. Current Snapshot: MvccSnapshot[applied={T|T < 7332635195291435008}]
I20260923 20:08:59.509730 4003 mvcc.cc:205] Tried to move back new op lower bound from 7332635195291435008 to 7332635195034107904. Current Snapshot: MvccSnapshot[applied={T|T < 7332635195291435008}]
I20260923 20:08:59.511162 3977 tablet_replica.cc:439] T 27fa16f0cfb047fd8779810f3c13977e P 98967392b7164818a63ed7e9912395c5: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "98967392b7164818a63ed7e9912395c5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98967392b7164818a63ed7e9912395c5" member_type: VOTER last_known_addr { host: "127.1.170.129" port: 44623 } } peers { permanent_uuid: "df7bc6713ea546638a209a6b211dfa0c" member_type: VOTER last_known_addr { host: "127.1.170.130" port: 39109 } } peers { permanent_uuid: "659357c7635644ea8fad67cfbd3c3c22" member_type: VOTER last_known_addr { host: "127.1.170.131" port: 37945 } } }
I20260923 20:08:59.511525 3977 tablet_replica.cc:442] T 27fa16f0cfb047fd8779810f3c13977e P 98967392b7164818a63ed7e9912395c5: This TxnStatusTablet replica's current role is: LEADER
I20260923 20:08:59.511780 3977 tablet_replica.cc:439] T 27fa16f0cfb047fd8779810f3c13977e P 98967392b7164818a63ed7e9912395c5: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "98967392b7164818a63ed7e9912395c5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98967392b7164818a63ed7e9912395c5" member_type: VOTER last_known_addr { host: "127.1.170.129" port: 44623 } } peers { permanent_uuid: "df7bc6713ea546638a209a6b211dfa0c" member_type: VOTER last_known_addr { host: "127.1.170.130" port: 39109 } } peers { permanent_uuid: "659357c7635644ea8fad67cfbd3c3c22" member_type: VOTER last_known_addr { host: "127.1.170.131" port: 37945 } } }
I20260923 20:08:59.512025 3977 tablet_replica.cc:442] T 27fa16f0cfb047fd8779810f3c13977e P 98967392b7164818a63ed7e9912395c5: This TxnStatusTablet replica's current role is: LEADER
I20260923 20:08:59.517802 3979 consensus_queue.cc:1035] T 906c58e7c4714f7bad7b4f568bfcebd7 P 659357c7635644ea8fad67cfbd3c3c22 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df7bc6713ea546638a209a6b211dfa0c" member_type: VOTER last_known_addr { host: "127.1.170.130" port: 39109 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 20:08:59.520391 3978 tablet_replica.cc:439] T 27fa16f0cfb047fd8779810f3c13977e P df7bc6713ea546638a209a6b211dfa0c: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "98967392b7164818a63ed7e9912395c5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98967392b7164818a63ed7e9912395c5" member_type: VOTER last_known_addr { host: "127.1.170.129" port: 44623 } } peers { permanent_uuid: "df7bc6713ea546638a209a6b211dfa0c" member_type: VOTER last_known_addr { host: "127.1.170.130" port: 39109 } } peers { permanent_uuid: "659357c7635644ea8fad67cfbd3c3c22" member_type: VOTER last_known_addr { host: "127.1.170.131" port: 37945 } } }
I20260923 20:08:59.520735 3978 tablet_replica.cc:442] T 27fa16f0cfb047fd8779810f3c13977e P df7bc6713ea546638a209a6b211dfa0c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260923 20:08:59.522121 4002 mvcc.cc:205] Tried to move back new op lower bound from 7332635195291435008 to 7332635195034107904. Current Snapshot: MvccSnapshot[applied={T|T < 7332635195291435008}]
I20260923 20:08:59.533465 3981 tablet_replica.cc:439] T 27fa16f0cfb047fd8779810f3c13977e P 659357c7635644ea8fad67cfbd3c3c22: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "98967392b7164818a63ed7e9912395c5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98967392b7164818a63ed7e9912395c5" member_type: VOTER last_known_addr { host: "127.1.170.129" port: 44623 } } peers { permanent_uuid: "df7bc6713ea546638a209a6b211dfa0c" member_type: VOTER last_known_addr { host: "127.1.170.130" port: 39109 } } peers { permanent_uuid: "659357c7635644ea8fad67cfbd3c3c22" member_type: VOTER last_known_addr { host: "127.1.170.131" port: 37945 } } }
I20260923 20:08:59.533763 3981 tablet_replica.cc:442] T 27fa16f0cfb047fd8779810f3c13977e P 659357c7635644ea8fad67cfbd3c3c22: This TxnStatusTablet replica's current role is: FOLLOWER
I20260923 20:08:59.542219 3992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:08:59.527806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33460 (local address 127.1.170.131:37945)
0923 20:08:59.527978 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0923 20:08:59.527984 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:08:59.535623 (+ 7639us) server_negotiation.cc:407] Connection header received
0923 20:08:59.535958 (+ 335us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:08:59.535964 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:08:59.536110 (+ 146us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:08:59.536249 (+ 139us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 20:08:59.536994 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:08:59.538081 (+ 1087us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:08:59.539498 (+ 1417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:08:59.539900 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:08:59.540066 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:08:59.540485 (+ 419us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 20:08:59.541207 (+ 722us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 20:08:59.541645 (+ 438us) server_negotiation.cc:1035] Waiting for connection context
0923 20:08:59.541808 (+ 163us) server_negotiation.cc:299] Negotiation successful
0923 20:08:59.541927 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260923 20:08:59.564065 3968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:08:59.556527 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.170.131:55081 (local address 127.1.170.129:44623)
0923 20:08:59.556789 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0923 20:08:59.556799 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0923 20:08:59.557268 (+ 469us) server_negotiation.cc:407] Connection header received
0923 20:08:59.557462 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:08:59.557467 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:08:59.557555 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:08:59.557654 (+ 99us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 20:08:59.558264 (+ 610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:08:59.559121 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:08:59.561237 (+ 2116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:08:59.562097 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:08:59.562260 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:08:59.562973 (+ 713us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 20:08:59.563139 (+ 166us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 20:08:59.563236 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0923 20:08:59.563717 (+ 481us) server_negotiation.cc:299] Negotiation successful
0923 20:08:59.563813 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260923 20:08:59.564299 4016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:08:59.556482 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.129:44623 (local address 127.1.170.131:55081)
0923 20:08:59.557126 (+ 644us) negotiation.cc:107] Waiting for socket to connect
0923 20:08:59.557146 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0923 20:08:59.557300 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:08:59.557692 (+ 392us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:08:59.557698 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:08:59.557712 (+ 14us) client_negotiation.cc:189] Negotiated authn=TOKEN
0923 20:08:59.558029 (+ 317us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:08:59.558041 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:08:59.559254 (+ 1213us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:08:59.559260 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:08:59.560974 (+ 1714us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:08:59.560989 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:08:59.562333 (+ 1344us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:08:59.562340 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:08:59.562534 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:08:59.562636 (+ 102us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0923 20:08:59.563316 (+ 680us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0923 20:08:59.563479 (+ 163us) client_negotiation.cc:769] Sending connection context
0923 20:08:59.563657 (+ 178us) client_negotiation.cc:240] Negotiation successful
0923 20:08:59.563819 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":445,"thread_start_us":331,"threads_started":1}
I20260923 20:08:59.611920 3992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:08:59.603176 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.170.129:47673 (local address 127.1.170.131:37945)
0923 20:08:59.603362 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0923 20:08:59.603369 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:08:59.604233 (+ 864us) server_negotiation.cc:407] Connection header received
0923 20:08:59.604540 (+ 307us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:08:59.604546 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:08:59.604642 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:08:59.604715 (+ 73us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 20:08:59.606139 (+ 1424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:08:59.607113 (+ 974us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:08:59.609397 (+ 2284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:08:59.610217 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:08:59.610378 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:08:59.611163 (+ 785us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 20:08:59.611288 (+ 125us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 20:08:59.611376 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0923 20:08:59.611595 (+ 219us) server_negotiation.cc:299] Negotiation successful
0923 20:08:59.611669 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260923 20:08:59.612162 4018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:08:59.603284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.131:37945 (local address 127.1.170.129:47673)
0923 20:08:59.604129 (+ 845us) negotiation.cc:107] Waiting for socket to connect
0923 20:08:59.604150 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0923 20:08:59.604384 (+ 234us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:08:59.604910 (+ 526us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:08:59.604914 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:08:59.604927 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0923 20:08:59.605713 (+ 786us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:08:59.605734 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:08:59.607277 (+ 1543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:08:59.607283 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:08:59.609184 (+ 1901us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:08:59.609200 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:08:59.610571 (+ 1371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:08:59.610579 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:08:59.610810 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:08:59.610901 (+ 91us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0923 20:08:59.611405 (+ 504us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0923 20:08:59.611417 (+ 12us) client_negotiation.cc:769] Sending connection context
0923 20:08:59.611515 (+ 98us) client_negotiation.cc:240] Negotiation successful
0923 20:08:59.611712 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":601,"thread_start_us":441,"threads_started":1}
I20260923 20:08:59.664896 3466 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34708:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 20:08:59.665203 3466 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34708:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 20:08:59.669508 3466 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4d74cac2e4f944208b16833320585dd4: Sending DeleteTablet for 3 replicas of tablet 906c58e7c4714f7bad7b4f568bfcebd7
I20260923 20:08:59.671231 3603 tablet_service.cc:1514] Processing DeleteTablet for tablet 906c58e7c4714f7bad7b4f568bfcebd7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 20:08:59 UTC) from {username='kudu', principal='kudu/127.1.170.190@KRBTEST.COM'} at 127.0.0.1:38920
I20260923 20:08:59.671939 3742 tablet_service.cc:1514] Processing DeleteTablet for tablet 906c58e7c4714f7bad7b4f568bfcebd7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 20:08:59 UTC) from {username='kudu', principal='kudu/127.1.170.190@KRBTEST.COM'} at 127.0.0.1:38888
I20260923 20:08:59.672013 3465 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34708:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 20:08:59.672736 3465 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 20:08:59.674672 3882 tablet_service.cc:1514] Processing DeleteTablet for tablet 906c58e7c4714f7bad7b4f568bfcebd7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 20:08:59 UTC) from {username='kudu', principal='kudu/127.1.170.190@KRBTEST.COM'} at 127.0.0.1:33456
I20260923 20:08:59.681684 3602 tablet_service.cc:1467] Processing CreateTablet for tablet 5dfdcebce09d479c914fa223f5de04b1 (DEFAULT_TABLE table=test-table [id=1aeac1127c8a48968497bdf25059109b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 20:08:59.681840 3741 tablet_service.cc:1467] Processing CreateTablet for tablet 5dfdcebce09d479c914fa223f5de04b1 (DEFAULT_TABLE table=test-table [id=1aeac1127c8a48968497bdf25059109b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 20:08:59.682289 3602 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5dfdcebce09d479c914fa223f5de04b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:08:59.682322 3741 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5dfdcebce09d479c914fa223f5de04b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:08:59.682636 4022 tablet_replica.cc:330] T 906c58e7c4714f7bad7b4f568bfcebd7 P df7bc6713ea546638a209a6b211dfa0c: stopping tablet replica
I20260923 20:08:59.683209 4022 raft_consensus.cc:2229] T 906c58e7c4714f7bad7b4f568bfcebd7 P df7bc6713ea546638a209a6b211dfa0c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 20:08:59.683712 4022 raft_consensus.cc:2258] T 906c58e7c4714f7bad7b4f568bfcebd7 P df7bc6713ea546638a209a6b211dfa0c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 20:08:59.685711 4022 ts_tablet_manager.cc:1905] T 906c58e7c4714f7bad7b4f568bfcebd7 P df7bc6713ea546638a209a6b211dfa0c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 20:08:59.686941 3881 tablet_service.cc:1467] Processing CreateTablet for tablet 5dfdcebce09d479c914fa223f5de04b1 (DEFAULT_TABLE table=test-table [id=1aeac1127c8a48968497bdf25059109b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 20:08:59.687505 3881 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5dfdcebce09d479c914fa223f5de04b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:08:59.703737 4022 ts_tablet_manager.cc:1918] T 906c58e7c4714f7bad7b4f568bfcebd7 P df7bc6713ea546638a209a6b211dfa0c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260923 20:08:59.703948 4022 log.cc:1196] T 906c58e7c4714f7bad7b4f568bfcebd7 P df7bc6713ea546638a209a6b211dfa0c: Deleting WAL directory at /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790194121577027-1706-0/minicluster-data/ts-1/wal/wals/906c58e7c4714f7bad7b4f568bfcebd7
I20260923 20:08:59.704519 4022 ts_tablet_manager.cc:1939] T 906c58e7c4714f7bad7b4f568bfcebd7 P df7bc6713ea546638a209a6b211dfa0c: Deleting consensus metadata
I20260923 20:08:59.707075 3451 catalog_manager.cc:4841] TS df7bc6713ea546638a209a6b211dfa0c (127.1.170.130:39109): tablet 906c58e7c4714f7bad7b4f568bfcebd7 (table test-table [id=4ad717d57bde40b3a63886d3cf533a37]) successfully deleted
I20260923 20:08:59.707918 4026 tablet_bootstrap.cc:492] T 5dfdcebce09d479c914fa223f5de04b1 P df7bc6713ea546638a209a6b211dfa0c: Bootstrap starting.
I20260923 20:08:59.708544 4025 tablet_bootstrap.cc:492] T 5dfdcebce09d479c914fa223f5de04b1 P 659357c7635644ea8fad67cfbd3c3c22: Bootstrap starting.
I20260923 20:08:59.710367 4026 tablet_bootstrap.cc:654] T 5dfdcebce09d479c914fa223f5de04b1 P df7bc6713ea546638a209a6b211dfa0c: Neither blocks nor log segments found. Creating new log.
I20260923 20:08:59.712363 4023 tablet_replica.cc:330] T 906c58e7c4714f7bad7b4f568bfcebd7 P 659357c7635644ea8fad67cfbd3c3c22: stopping tablet replica
I20260923 20:08:59.712363 4024 tablet_bootstrap.cc:492] T 5dfdcebce09d479c914fa223f5de04b1 P 98967392b7164818a63ed7e9912395c5: Bootstrap starting.
I20260923 20:08:59.712833 4025 tablet_bootstrap.cc:654] T 5dfdcebce09d479c914fa223f5de04b1 P 659357c7635644ea8fad67cfbd3c3c22: Neither blocks nor log segments found. Creating new log.
I20260923 20:08:59.712853 4023 raft_consensus.cc:2229] T 906c58e7c4714f7bad7b4f568bfcebd7 P 659357c7635644ea8fad67cfbd3c3c22 [term 1 LEADER]: Raft consensus shutting down.
I20260923 20:08:59.713368 4021 tablet_replica.cc:330] T 906c58e7c4714f7bad7b4f568bfcebd7 P 98967392b7164818a63ed7e9912395c5: stopping tablet replica
I20260923 20:08:59.713822 4021 raft_consensus.cc:2229] T 906c58e7c4714f7bad7b4f568bfcebd7 P 98967392b7164818a63ed7e9912395c5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 20:08:59.713809 4023 raft_consensus.cc:2258] T 906c58e7c4714f7bad7b4f568bfcebd7 P 659357c7635644ea8fad67cfbd3c3c22 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 20:08:59.714131 4021 raft_consensus.cc:2258] T 906c58e7c4714f7bad7b4f568bfcebd7 P 98967392b7164818a63ed7e9912395c5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 20:08:59.716288 4023 ts_tablet_manager.cc:1905] T 906c58e7c4714f7bad7b4f568bfcebd7 P 659357c7635644ea8fad67cfbd3c3c22: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 20:08:59.718794 4024 tablet_bootstrap.cc:654] T 5dfdcebce09d479c914fa223f5de04b1 P 98967392b7164818a63ed7e9912395c5: Neither blocks nor log segments found. Creating new log.
I20260923 20:08:59.721328 4026 tablet_bootstrap.cc:492] T 5dfdcebce09d479c914fa223f5de04b1 P df7bc6713ea546638a209a6b211dfa0c: No bootstrap required, opened a new log
I20260923 20:08:59.721527 4026 ts_tablet_manager.cc:1397] T 5dfdcebce09d479c914fa223f5de04b1 P df7bc6713ea546638a209a6b211dfa0c: Time spent bootstrapping tablet: real 0.014s user 0.003s sys 0.000s
I20260923 20:08:59.722433 4026 raft_consensus.cc:348] T 5dfdcebce09d479c914fa223f5de04b1 P df7bc6713ea546638a209a6b211dfa0c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98967392b7164818a63ed7e9912395c5" member_type: VOTER last_known_addr { host: "127.1.170.129" port: 44623 } } peers { permanent_uuid: "df7bc6713ea546638a209a6b211dfa0c" member_type: VOTER last_known_addr { host: "127.1.170.130" port: 39109 } } peers { permanent_uuid: "659357c7635644ea8fad67cfbd3c3c22" member_type: VOTER last_known_addr { host: "127.1.170.131" port: 37945 } }
I20260923 20:08:59.722661 4026 raft_consensus.cc:374] T 5dfdcebce09d479c914fa223f5de04b1 P df7bc6713ea546638a209a6b211dfa0c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:08:59.722754 4026 raft_consensus.cc:729] T 5dfdcebce09d479c914fa223f5de04b1 P df7bc6713ea546638a209a6b211dfa0c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df7bc6713ea546638a209a6b211dfa0c, State: Initialized, Role: FOLLOWER
I20260923 20:08:59.723327 4026 consensus_queue.cc:260] T 5dfdcebce09d479c914fa223f5de04b1 P df7bc6713ea546638a209a6b211dfa0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98967392b7164818a63ed7e9912395c5" member_type: VOTER last_known_addr { host: "127.1.170.129" port: 44623 } } peers { permanent_uuid: "df7bc6713ea546638a209a6b211dfa0c" member_type: VOTER last_known_addr { host: "127.1.170.130" port: 39109 } } peers { permanent_uuid: "659357c7635644ea8fad67cfbd3c3c22" member_type: VOTER last_known_addr { host: "127.1.170.131" port: 37945 } }
I20260923 20:08:59.724023 4026 ts_tablet_manager.cc:1428] T 5dfdcebce09d479c914fa223f5de04b1 P df7bc6713ea546638a209a6b211dfa0c: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260923 20:08:59.724489 4025 tablet_bootstrap.cc:492] T 5dfdcebce09d479c914fa223f5de04b1 P 659357c7635644ea8fad67cfbd3c3c22: No bootstrap required, opened a new log
I20260923 20:08:59.724675 4025 ts_tablet_manager.cc:1397] T 5dfdcebce09d479c914fa223f5de04b1 P 659357c7635644ea8fad67cfbd3c3c22: Time spent bootstrapping tablet: real 0.016s user 0.003s sys 0.008s
I20260923 20:08:59.725730 4025 raft_consensus.cc:348] T 5dfdcebce09d479c914fa223f5de04b1 P 659357c7635644ea8fad67cfbd3c3c22 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98967392b7164818a63ed7e9912395c5" member_type: VOTER last_known_addr { host: "127.1.170.129" port: 44623 } } peers { permanent_uuid: "df7bc6713ea546638a209a6b211dfa0c" member_type: VOTER last_known_addr { host: "127.1.170.130" port: 39109 } } peers { permanent_uuid: "659357c7635644ea8fad67cfbd3c3c22" member_type: VOTER last_known_addr { host: "127.1.170.131" port: 37945 } }
I20260923 20:08:59.725975 4023 ts_tablet_manager.cc:1918] T 906c58e7c4714f7bad7b4f568bfcebd7 P 659357c7635644ea8fad67cfbd3c3c22: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260923 20:08:59.726115 4025 raft_consensus.cc:374] T 5dfdcebce09d479c914fa223f5de04b1 P 659357c7635644ea8fad67cfbd3c3c22 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:08:59.726228 4023 log.cc:1196] T 906c58e7c4714f7bad7b4f568bfcebd7 P 659357c7635644ea8fad67cfbd3c3c22: Deleting WAL directory at /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790194121577027-1706-0/minicluster-data/ts-2/wal/wals/906c58e7c4714f7bad7b4f568bfcebd7
I20260923 20:08:59.726307 4025 raft_consensus.cc:729] T 5dfdcebce09d479c914fa223f5de04b1 P 659357c7635644ea8fad67cfbd3c3c22 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 659357c7635644ea8fad67cfbd3c3c22, State: Initialized, Role: FOLLOWER
I20260923 20:08:59.726843 4025 consensus_queue.cc:260] T 5dfdcebce09d479c914fa223f5de04b1 P 659357c7635644ea8fad67cfbd3c3c22 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98967392b7164818a63ed7e9912395c5" member_type: VOTER last_known_addr { host: "127.1.170.129" port: 44623 } } peers { permanent_uuid: "df7bc6713ea546638a209a6b211dfa0c" member_type: VOTER last_known_addr { host: "127.1.170.130" port: 39109 } } peers { permanent_uuid: "659357c7635644ea8fad67cfbd3c3c22" member_type: VOTER last_known_addr { host: "127.1.170.131" port: 37945 } }
I20260923 20:08:59.727591 4023 ts_tablet_manager.cc:1939] T 906c58e7c4714f7bad7b4f568bfcebd7 P 659357c7635644ea8fad67cfbd3c3c22: Deleting consensus metadata
I20260923 20:08:59.728494 4025 ts_tablet_manager.cc:1428] T 5dfdcebce09d479c914fa223f5de04b1 P 659357c7635644ea8fad67cfbd3c3c22: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20260923 20:08:59.728488 4021 ts_tablet_manager.cc:1905] T 906c58e7c4714f7bad7b4f568bfcebd7 P 98967392b7164818a63ed7e9912395c5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 20:08:59.729347 3453 catalog_manager.cc:4841] TS 659357c7635644ea8fad67cfbd3c3c22 (127.1.170.131:37945): tablet 906c58e7c4714f7bad7b4f568bfcebd7 (table test-table [id=4ad717d57bde40b3a63886d3cf533a37]) successfully deleted
I20260923 20:08:59.735669 4024 tablet_bootstrap.cc:492] T 5dfdcebce09d479c914fa223f5de04b1 P 98967392b7164818a63ed7e9912395c5: No bootstrap required, opened a new log
I20260923 20:08:59.735872 4024 ts_tablet_manager.cc:1397] T 5dfdcebce09d479c914fa223f5de04b1 P 98967392b7164818a63ed7e9912395c5: Time spent bootstrapping tablet: real 0.024s user 0.003s sys 0.004s
I20260923 20:08:59.736778 4024 raft_consensus.cc:348] T 5dfdcebce09d479c914fa223f5de04b1 P 98967392b7164818a63ed7e9912395c5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98967392b7164818a63ed7e9912395c5" member_type: VOTER last_known_addr { host: "127.1.170.129" port: 44623 } } peers { permanent_uuid: "df7bc6713ea546638a209a6b211dfa0c" member_type: VOTER last_known_addr { host: "127.1.170.130" port: 39109 } } peers { permanent_uuid: "659357c7635644ea8fad67cfbd3c3c22" member_type: VOTER last_known_addr { host: "127.1.170.131" port: 37945 } }
I20260923 20:08:59.737074 4024 raft_consensus.cc:374] T 5dfdcebce09d479c914fa223f5de04b1 P 98967392b7164818a63ed7e9912395c5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:08:59.737193 4024 raft_consensus.cc:729] T 5dfdcebce09d479c914fa223f5de04b1 P 98967392b7164818a63ed7e9912395c5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98967392b7164818a63ed7e9912395c5, State: Initialized, Role: FOLLOWER
I20260923 20:08:59.737485 4024 consensus_queue.cc:260] T 5dfdcebce09d479c914fa223f5de04b1 P 98967392b7164818a63ed7e9912395c5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98967392b7164818a63ed7e9912395c5" member_type: VOTER last_known_addr { host: "127.1.170.129" port: 44623 } } peers { permanent_uuid: "df7bc6713ea546638a209a6b211dfa0c" member_type: VOTER last_known_addr { host: "127.1.170.130" port: 39109 } } peers { permanent_uuid: "659357c7635644ea8fad67cfbd3c3c22" member_type: VOTER last_known_addr { host: "127.1.170.131" port: 37945 } }
I20260923 20:08:59.738193 4024 ts_tablet_manager.cc:1428] T 5dfdcebce09d479c914fa223f5de04b1 P 98967392b7164818a63ed7e9912395c5: Time spent starting tablet: real 0.002s user 0.002s sys 0.002s
I20260923 20:08:59.747535 4021 ts_tablet_manager.cc:1918] T 906c58e7c4714f7bad7b4f568bfcebd7 P 98967392b7164818a63ed7e9912395c5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260923 20:08:59.747741 4021 log.cc:1196] T 906c58e7c4714f7bad7b4f568bfcebd7 P 98967392b7164818a63ed7e9912395c5: Deleting WAL directory at /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790194121577027-1706-0/minicluster-data/ts-0/wal/wals/906c58e7c4714f7bad7b4f568bfcebd7
I20260923 20:08:59.748292 4021 ts_tablet_manager.cc:1939] T 906c58e7c4714f7bad7b4f568bfcebd7 P 98967392b7164818a63ed7e9912395c5: Deleting consensus metadata
I20260923 20:08:59.750182 3451 catalog_manager.cc:4841] TS 98967392b7164818a63ed7e9912395c5 (127.1.170.129:44623): tablet 906c58e7c4714f7bad7b4f568bfcebd7 (table test-table [id=4ad717d57bde40b3a63886d3cf533a37]) successfully deleted
I20260923 20:08:59.945062 3978 raft_consensus.cc:482] T 5dfdcebce09d479c914fa223f5de04b1 P df7bc6713ea546638a209a6b211dfa0c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 20:08:59.945298 3978 raft_consensus.cc:504] T 5dfdcebce09d479c914fa223f5de04b1 P df7bc6713ea546638a209a6b211dfa0c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98967392b7164818a63ed7e9912395c5" member_type: VOTER last_known_addr { host: "127.1.170.129" port: 44623 } } peers { permanent_uuid: "df7bc6713ea546638a209a6b211dfa0c" member_type: VOTER last_known_addr { host: "127.1.170.130" port: 39109 } } peers { permanent_uuid: "659357c7635644ea8fad67cfbd3c3c22" member_type: VOTER last_known_addr { host: "127.1.170.131" port: 37945 } }
I20260923 20:08:59.947186 3978 leader_election.cc:288] T 5dfdcebce09d479c914fa223f5de04b1 P df7bc6713ea546638a209a6b211dfa0c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 98967392b7164818a63ed7e9912395c5 (127.1.170.129:44623), 659357c7635644ea8fad67cfbd3c3c22 (127.1.170.131:37945)
I20260923 20:08:59.954316 4010 raft_consensus.cc:482] T 5dfdcebce09d479c914fa223f5de04b1 P 659357c7635644ea8fad67cfbd3c3c22 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 20:08:59.954577 4010 raft_consensus.cc:504] T 5dfdcebce09d479c914fa223f5de04b1 P 659357c7635644ea8fad67cfbd3c3c22 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98967392b7164818a63ed7e9912395c5" member_type: VOTER last_known_addr { host: "127.1.170.129" port: 44623 } } peers { permanent_uuid: "df7bc6713ea546638a209a6b211dfa0c" member_type: VOTER last_known_addr { host: "127.1.170.130" port: 39109 } } peers { permanent_uuid: "659357c7635644ea8fad67cfbd3c3c22" member_type: VOTER last_known_addr { host: "127.1.170.131" port: 37945 } }
I20260923 20:08:59.955453 4010 leader_election.cc:288] T 5dfdcebce09d479c914fa223f5de04b1 P 659357c7635644ea8fad67cfbd3c3c22 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 98967392b7164818a63ed7e9912395c5 (127.1.170.129:44623), df7bc6713ea546638a209a6b211dfa0c (127.1.170.130:39109)
I20260923 20:08:59.956310 3628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5dfdcebce09d479c914fa223f5de04b1" candidate_uuid: "659357c7635644ea8fad67cfbd3c3c22" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98967392b7164818a63ed7e9912395c5" is_pre_election: true
I20260923 20:08:59.956578 3628 raft_consensus.cc:2454] T 5dfdcebce09d479c914fa223f5de04b1 P 98967392b7164818a63ed7e9912395c5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 659357c7635644ea8fad67cfbd3c3c22 in term 0.
I20260923 20:08:59.957186 3761 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5dfdcebce09d479c914fa223f5de04b1" candidate_uuid: "659357c7635644ea8fad67cfbd3c3c22" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df7bc6713ea546638a209a6b211dfa0c" is_pre_election: true
I20260923 20:08:59.957436 3761 raft_consensus.cc:2454] T 5dfdcebce09d479c914fa223f5de04b1 P df7bc6713ea546638a209a6b211dfa0c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 659357c7635644ea8fad67cfbd3c3c22 in term 0.
I20260923 20:08:59.958110 3833 leader_election.cc:302] T 5dfdcebce09d479c914fa223f5de04b1 P 659357c7635644ea8fad67cfbd3c3c22 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 659357c7635644ea8fad67cfbd3c3c22, df7bc6713ea546638a209a6b211dfa0c; no voters:
I20260923 20:08:59.958580 4010 raft_consensus.cc:2781] T 5dfdcebce09d479c914fa223f5de04b1 P 659357c7635644ea8fad67cfbd3c3c22 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 20:08:59.958699 4010 raft_consensus.cc:482] T 5dfdcebce09d479c914fa223f5de04b1 P 659357c7635644ea8fad67cfbd3c3c22 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 20:08:59.958786 4010 raft_consensus.cc:3037] T 5dfdcebce09d479c914fa223f5de04b1 P 659357c7635644ea8fad67cfbd3c3c22 [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:08:59.962423 4010 raft_consensus.cc:504] T 5dfdcebce09d479c914fa223f5de04b1 P 659357c7635644ea8fad67cfbd3c3c22 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98967392b7164818a63ed7e9912395c5" member_type: VOTER last_known_addr { host: "127.1.170.129" port: 44623 } } peers { permanent_uuid: "df7bc6713ea546638a209a6b211dfa0c" member_type: VOTER last_known_addr { host: "127.1.170.130" port: 39109 } } peers { permanent_uuid: "659357c7635644ea8fad67cfbd3c3c22" member_type: VOTER last_known_addr { host: "127.1.170.131" port: 37945 } }
I20260923 20:08:59.963505 3628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5dfdcebce09d479c914fa223f5de04b1" candidate_uuid: "659357c7635644ea8fad67cfbd3c3c22" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98967392b7164818a63ed7e9912395c5"
I20260923 20:08:59.963794 3628 raft_consensus.cc:3037] T 5dfdcebce09d479c914fa223f5de04b1 P 98967392b7164818a63ed7e9912395c5 [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:08:59.967180 3628 raft_consensus.cc:2454] T 5dfdcebce09d479c914fa223f5de04b1 P 98967392b7164818a63ed7e9912395c5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 659357c7635644ea8fad67cfbd3c3c22 in term 1.
I20260923 20:08:59.967703 3833 leader_election.cc:302] T 5dfdcebce09d479c914fa223f5de04b1 P 659357c7635644ea8fad67cfbd3c3c22 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 659357c7635644ea8fad67cfbd3c3c22, 98967392b7164818a63ed7e9912395c5; no voters:
I20260923 20:08:59.968108 3979 raft_consensus.cc:2781] T 5dfdcebce09d479c914fa223f5de04b1 P 659357c7635644ea8fad67cfbd3c3c22 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 20:08:59.968294 3979 raft_consensus.cc:686] T 5dfdcebce09d479c914fa223f5de04b1 P 659357c7635644ea8fad67cfbd3c3c22 [term 1 LEADER]: Becoming Leader. State: Replica: 659357c7635644ea8fad67cfbd3c3c22, State: Running, Role: LEADER
I20260923 20:08:59.968367 3968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:08:59.948163 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.170.130:36481 (local address 127.1.170.129:44623)
0923 20:08:59.948353 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0923 20:08:59.948359 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:08:59.959575 (+ 11216us) server_negotiation.cc:407] Connection header received
0923 20:08:59.960028 (+ 453us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:08:59.960034 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:08:59.960114 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:08:59.960177 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:08:59.960902 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:08:59.962087 (+ 1185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:08:59.965065 (+ 2978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:08:59.966616 (+ 1551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:08:59.966806 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:08:59.966889 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0923 20:08:59.967882 (+ 993us) server_negotiation.cc:299] Negotiation successful
0923 20:08:59.968045 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260923 20:08:59.968567 3979 consensus_queue.cc:237] T 5dfdcebce09d479c914fa223f5de04b1 P 659357c7635644ea8fad67cfbd3c3c22 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98967392b7164818a63ed7e9912395c5" member_type: VOTER last_known_addr { host: "127.1.170.129" port: 44623 } } peers { permanent_uuid: "df7bc6713ea546638a209a6b211dfa0c" member_type: VOTER last_known_addr { host: "127.1.170.130" port: 39109 } } peers { permanent_uuid: "659357c7635644ea8fad67cfbd3c3c22" member_type: VOTER last_known_addr { host: "127.1.170.131" port: 37945 } }
I20260923 20:08:59.969491 4031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:08:59.947524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.129:44623 (local address 127.1.170.130:36481)
0923 20:08:59.959457 (+ 11933us) negotiation.cc:107] Waiting for socket to connect
0923 20:08:59.959479 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0923 20:08:59.959852 (+ 373us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:08:59.960375 (+ 523us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:08:59.960380 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:08:59.960416 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:08:59.960669 (+ 253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:08:59.960680 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:08:59.962285 (+ 1605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:08:59.962290 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:08:59.964849 (+ 2559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:08:59.964869 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:08:59.967438 (+ 2569us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:08:59.967445 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:08:59.967633 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:08:59.967673 (+ 40us) client_negotiation.cc:769] Sending connection context
0923 20:08:59.968919 (+ 1246us) client_negotiation.cc:240] Negotiation successful
0923 20:08:59.969094 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11801,"threads_started":1}
I20260923 20:08:59.972038 3761 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5dfdcebce09d479c914fa223f5de04b1" candidate_uuid: "659357c7635644ea8fad67cfbd3c3c22" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df7bc6713ea546638a209a6b211dfa0c"
I20260923 20:08:59.972249 3761 raft_consensus.cc:3037] T 5dfdcebce09d479c914fa223f5de04b1 P df7bc6713ea546638a209a6b211dfa0c [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:08:59.975320 3761 raft_consensus.cc:2454] T 5dfdcebce09d479c914fa223f5de04b1 P df7bc6713ea546638a209a6b211dfa0c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 659357c7635644ea8fad67cfbd3c3c22 in term 1.
I20260923 20:08:59.976020 4010 leader_election.cc:288] T 5dfdcebce09d479c914fa223f5de04b1 P 659357c7635644ea8fad67cfbd3c3c22 [CANDIDATE]: Term 1 election: Requested vote from peers 98967392b7164818a63ed7e9912395c5 (127.1.170.129:44623), df7bc6713ea546638a209a6b211dfa0c (127.1.170.130:39109)
I20260923 20:08:59.977643 3992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:08:59.947913 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.170.130:60659 (local address 127.1.170.131:37945)
0923 20:08:59.948086 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0923 20:08:59.948092 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:08:59.970058 (+ 21966us) server_negotiation.cc:407] Connection header received
0923 20:08:59.970395 (+ 337us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:08:59.970402 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:08:59.970474 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:08:59.970533 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:08:59.971086 (+ 553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:08:59.972226 (+ 1140us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:08:59.974758 (+ 2532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:08:59.976260 (+ 1502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:08:59.976400 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:08:59.976482 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0923 20:08:59.977141 (+ 659us) server_negotiation.cc:299] Negotiation successful
0923 20:08:59.977292 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260923 20:08:59.978597 4031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:08:59.947812 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.131:37945 (local address 127.1.170.130:60659)
0923 20:08:59.969960 (+ 22148us) negotiation.cc:107] Waiting for socket to connect
0923 20:08:59.969982 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0923 20:08:59.970178 (+ 196us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:08:59.970709 (+ 531us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:08:59.970713 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:08:59.970744 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:08:59.970929 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:08:59.970940 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:08:59.972483 (+ 1543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:08:59.972488 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:08:59.974558 (+ 2070us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:08:59.974573 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:08:59.976719 (+ 2146us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:08:59.976725 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:08:59.976946 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:08:59.976980 (+ 34us) client_negotiation.cc:769] Sending connection context
0923 20:08:59.978082 (+ 1102us) client_negotiation.cc:240] Negotiation successful
0923 20:08:59.978244 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":22023,"thread_start_us":11778,"threads_started":1}
I20260923 20:08:59.979439 3902 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5dfdcebce09d479c914fa223f5de04b1" candidate_uuid: "df7bc6713ea546638a209a6b211dfa0c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "659357c7635644ea8fad67cfbd3c3c22" is_pre_election: true
I20260923 20:08:59.982499 3465 catalog_manager.cc:5495] T 5dfdcebce09d479c914fa223f5de04b1 P 659357c7635644ea8fad67cfbd3c3c22 reported cstate change: term changed from 0 to 1, leader changed from <none> to 659357c7635644ea8fad67cfbd3c3c22 (127.1.170.131). New cstate: current_term: 1 leader_uuid: "659357c7635644ea8fad67cfbd3c3c22" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98967392b7164818a63ed7e9912395c5" member_type: VOTER last_known_addr { host: "127.1.170.129" port: 44623 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df7bc6713ea546638a209a6b211dfa0c" member_type: VOTER last_known_addr { host: "127.1.170.130" port: 39109 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "659357c7635644ea8fad67cfbd3c3c22" member_type: VOTER last_known_addr { host: "127.1.170.131" port: 37945 } health_report { overall_health: HEALTHY } } }
I20260923 20:08:59.984387 3628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5dfdcebce09d479c914fa223f5de04b1" candidate_uuid: "df7bc6713ea546638a209a6b211dfa0c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98967392b7164818a63ed7e9912395c5" is_pre_election: true
I20260923 20:08:59.984856 3628 raft_consensus.cc:2379] T 5dfdcebce09d479c914fa223f5de04b1 P 98967392b7164818a63ed7e9912395c5 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df7bc6713ea546638a209a6b211dfa0c in current term 1: Already voted for candidate 659357c7635644ea8fad67cfbd3c3c22 in this term.
I20260923 20:08:59.985916 3694 leader_election.cc:302] T 5dfdcebce09d479c914fa223f5de04b1 P df7bc6713ea546638a209a6b211dfa0c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df7bc6713ea546638a209a6b211dfa0c; no voters: 659357c7635644ea8fad67cfbd3c3c22, 98967392b7164818a63ed7e9912395c5
I20260923 20:08:59.986518 3978 raft_consensus.cc:2726] T 5dfdcebce09d479c914fa223f5de04b1 P df7bc6713ea546638a209a6b211dfa0c [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260923 20:09:00.015388 3762 raft_consensus.cc:1261] T 5dfdcebce09d479c914fa223f5de04b1 P df7bc6713ea546638a209a6b211dfa0c [term 1 FOLLOWER]: Refusing update from remote peer 659357c7635644ea8fad67cfbd3c3c22: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 20:09:00.015486 3628 raft_consensus.cc:1261] T 5dfdcebce09d479c914fa223f5de04b1 P 98967392b7164818a63ed7e9912395c5 [term 1 FOLLOWER]: Refusing update from remote peer 659357c7635644ea8fad67cfbd3c3c22: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 20:09:00.016012 4010 consensus_queue.cc:1035] T 5dfdcebce09d479c914fa223f5de04b1 P 659357c7635644ea8fad67cfbd3c3c22 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df7bc6713ea546638a209a6b211dfa0c" member_type: VOTER last_known_addr { host: "127.1.170.130" port: 39109 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 20:09:00.016618 3979 consensus_queue.cc:1035] T 5dfdcebce09d479c914fa223f5de04b1 P 659357c7635644ea8fad67cfbd3c3c22 [LEADER]: Connected to new peer: Peer: permanent_uuid: "98967392b7164818a63ed7e9912395c5" member_type: VOTER last_known_addr { host: "127.1.170.129" port: 44623 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260923 20:09:00.044786 4003 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260923 20:09:01.639952 3465 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34708:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 20:09:01.640178 3465 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34708:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 20:09:01.644778 3465 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4d74cac2e4f944208b16833320585dd4: Sending DeleteTablet for 3 replicas of tablet 5dfdcebce09d479c914fa223f5de04b1
I20260923 20:09:01.645741 3603 tablet_service.cc:1514] Processing DeleteTablet for tablet 5dfdcebce09d479c914fa223f5de04b1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 20:09:01 UTC) from {username='kudu', principal='kudu/127.1.170.190@KRBTEST.COM'} at 127.0.0.1:38920
I20260923 20:09:01.645813 3882 tablet_service.cc:1514] Processing DeleteTablet for tablet 5dfdcebce09d479c914fa223f5de04b1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 20:09:01 UTC) from {username='kudu', principal='kudu/127.1.170.190@KRBTEST.COM'} at 127.0.0.1:33456
I20260923 20:09:01.645805 3741 tablet_service.cc:1514] Processing DeleteTablet for tablet 5dfdcebce09d479c914fa223f5de04b1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 20:09:01 UTC) from {username='kudu', principal='kudu/127.1.170.190@KRBTEST.COM'} at 127.0.0.1:38888
I20260923 20:09:01.656780 4056 tablet_replica.cc:330] T 5dfdcebce09d479c914fa223f5de04b1 P 98967392b7164818a63ed7e9912395c5: stopping tablet replica
I20260923 20:09:01.657274 4056 raft_consensus.cc:2229] T 5dfdcebce09d479c914fa223f5de04b1 P 98967392b7164818a63ed7e9912395c5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 20:09:01.657641 4056 raft_consensus.cc:2258] T 5dfdcebce09d479c914fa223f5de04b1 P 98967392b7164818a63ed7e9912395c5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 20:09:01.659075 4056 ts_tablet_manager.cc:1905] T 5dfdcebce09d479c914fa223f5de04b1 P 98967392b7164818a63ed7e9912395c5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 20:09:01.661146 4057 tablet_replica.cc:330] T 5dfdcebce09d479c914fa223f5de04b1 P 659357c7635644ea8fad67cfbd3c3c22: stopping tablet replica
I20260923 20:09:01.661608 4058 tablet_replica.cc:330] T 5dfdcebce09d479c914fa223f5de04b1 P df7bc6713ea546638a209a6b211dfa0c: stopping tablet replica
I20260923 20:09:01.661810 4058 raft_consensus.cc:2229] T 5dfdcebce09d479c914fa223f5de04b1 P df7bc6713ea546638a209a6b211dfa0c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 20:09:01.662079 4058 raft_consensus.cc:2258] T 5dfdcebce09d479c914fa223f5de04b1 P df7bc6713ea546638a209a6b211dfa0c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 20:09:01.662158 4057 raft_consensus.cc:2229] T 5dfdcebce09d479c914fa223f5de04b1 P 659357c7635644ea8fad67cfbd3c3c22 [term 1 LEADER]: Raft consensus shutting down.
I20260923 20:09:01.662539 4057 raft_consensus.cc:2258] T 5dfdcebce09d479c914fa223f5de04b1 P 659357c7635644ea8fad67cfbd3c3c22 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 20:09:01.663388 4058 ts_tablet_manager.cc:1905] T 5dfdcebce09d479c914fa223f5de04b1 P df7bc6713ea546638a209a6b211dfa0c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 20:09:01.663571 4057 ts_tablet_manager.cc:1905] T 5dfdcebce09d479c914fa223f5de04b1 P 659357c7635644ea8fad67cfbd3c3c22: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 20:09:01.666608 4056 ts_tablet_manager.cc:1918] T 5dfdcebce09d479c914fa223f5de04b1 P 98967392b7164818a63ed7e9912395c5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260923 20:09:01.666848 4056 log.cc:1196] T 5dfdcebce09d479c914fa223f5de04b1 P 98967392b7164818a63ed7e9912395c5: Deleting WAL directory at /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790194121577027-1706-0/minicluster-data/ts-0/wal/wals/5dfdcebce09d479c914fa223f5de04b1
I20260923 20:09:01.667380 4056 ts_tablet_manager.cc:1939] T 5dfdcebce09d479c914fa223f5de04b1 P 98967392b7164818a63ed7e9912395c5: Deleting consensus metadata
I20260923 20:09:01.668666 3451 catalog_manager.cc:4841] TS 98967392b7164818a63ed7e9912395c5 (127.1.170.129:44623): tablet 5dfdcebce09d479c914fa223f5de04b1 (table test-table [id=1aeac1127c8a48968497bdf25059109b]) successfully deleted
I20260923 20:09:01.669967 4058 ts_tablet_manager.cc:1918] T 5dfdcebce09d479c914fa223f5de04b1 P df7bc6713ea546638a209a6b211dfa0c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260923 20:09:01.670146 4058 log.cc:1196] T 5dfdcebce09d479c914fa223f5de04b1 P df7bc6713ea546638a209a6b211dfa0c: Deleting WAL directory at /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790194121577027-1706-0/minicluster-data/ts-1/wal/wals/5dfdcebce09d479c914fa223f5de04b1
I20260923 20:09:01.670675 4058 ts_tablet_manager.cc:1939] T 5dfdcebce09d479c914fa223f5de04b1 P df7bc6713ea546638a209a6b211dfa0c: Deleting consensus metadata
I20260923 20:09:01.670849 4057 ts_tablet_manager.cc:1918] T 5dfdcebce09d479c914fa223f5de04b1 P 659357c7635644ea8fad67cfbd3c3c22: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260923 20:09:01.670990 4057 log.cc:1196] T 5dfdcebce09d479c914fa223f5de04b1 P 659357c7635644ea8fad67cfbd3c3c22: Deleting WAL directory at /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790194121577027-1706-0/minicluster-data/ts-2/wal/wals/5dfdcebce09d479c914fa223f5de04b1
I20260923 20:09:01.671411 4057 ts_tablet_manager.cc:1939] T 5dfdcebce09d479c914fa223f5de04b1 P 659357c7635644ea8fad67cfbd3c3c22: Deleting consensus metadata
I20260923 20:09:01.671778 3451 catalog_manager.cc:4841] TS df7bc6713ea546638a209a6b211dfa0c (127.1.170.130:39109): tablet 5dfdcebce09d479c914fa223f5de04b1 (table test-table [id=1aeac1127c8a48968497bdf25059109b]) successfully deleted
I20260923 20:09:01.672539 3453 catalog_manager.cc:4841] TS 659357c7635644ea8fad67cfbd3c3c22 (127.1.170.131:37945): tablet 5dfdcebce09d479c914fa223f5de04b1 (table test-table [id=1aeac1127c8a48968497bdf25059109b]) successfully deleted
Sep 23 20:09:01 dist-test-slave-th4g krb5kdc[3421](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194141, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 23 20:09:01 dist-test-slave-th4g krb5kdc[3421](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194141, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.1.170.190@KRBTEST.COM
I20260923 20:09:01.707494 4066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:09:01.692216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34722 (local address 127.1.170.190:35033)
0923 20:09:01.692955 (+ 739us) server_negotiation.cc:206] Beginning negotiation
0923 20:09:01.692962 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:09:01.692995 (+ 33us) server_negotiation.cc:407] Connection header received
0923 20:09:01.693067 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:09:01.693072 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:09:01.693176 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:09:01.693274 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:09:01.694804 (+ 1530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:09:01.695580 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:09:01.696896 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:09:01.697326 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:09:01.697468 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:09:01.701511 (+ 4043us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:09:01.701516 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:09:01.701520 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:09:01.701555 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:09:01.704871 (+ 3316us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:09:01.705565 (+ 694us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:09:01.705572 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:09:01.705576 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:09:01.705693 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:09:01.706114 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:09:01.706119 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:09:01.706121 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:09:01.706392 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:09:01.706628 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0923 20:09:01.706923 (+ 295us) server_negotiation.cc:299] Negotiation successful
0923 20:09:01.707133 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":597,"thread_start_us":448,"threads_started":1}
I20260923 20:09:01.711289 3465 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34722:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 20:09:01.711918 3465 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 20:09:01.718760 3603 tablet_service.cc:1467] Processing CreateTablet for tablet 789f118b81e04ad289cf4b4f9c9ecd85 (DEFAULT_TABLE table=test-table [id=b7cbd14a138b44fe8c860a3469b27bf8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 20:09:01.718848 3741 tablet_service.cc:1467] Processing CreateTablet for tablet 789f118b81e04ad289cf4b4f9c9ecd85 (DEFAULT_TABLE table=test-table [id=b7cbd14a138b44fe8c860a3469b27bf8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 20:09:01.718941 3882 tablet_service.cc:1467] Processing CreateTablet for tablet 789f118b81e04ad289cf4b4f9c9ecd85 (DEFAULT_TABLE table=test-table [id=b7cbd14a138b44fe8c860a3469b27bf8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 20:09:01.719215 3603 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 789f118b81e04ad289cf4b4f9c9ecd85. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:09:01.719309 3882 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 789f118b81e04ad289cf4b4f9c9ecd85. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:09:01.719311 3741 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 789f118b81e04ad289cf4b4f9c9ecd85. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:09:01.725989 4067 tablet_bootstrap.cc:492] T 789f118b81e04ad289cf4b4f9c9ecd85 P df7bc6713ea546638a209a6b211dfa0c: Bootstrap starting.
I20260923 20:09:01.726626 4068 tablet_bootstrap.cc:492] T 789f118b81e04ad289cf4b4f9c9ecd85 P 98967392b7164818a63ed7e9912395c5: Bootstrap starting.
I20260923 20:09:01.727134 4069 tablet_bootstrap.cc:492] T 789f118b81e04ad289cf4b4f9c9ecd85 P 659357c7635644ea8fad67cfbd3c3c22: Bootstrap starting.
I20260923 20:09:01.728550 4067 tablet_bootstrap.cc:654] T 789f118b81e04ad289cf4b4f9c9ecd85 P df7bc6713ea546638a209a6b211dfa0c: Neither blocks nor log segments found. Creating new log.
I20260923 20:09:01.728708 4069 tablet_bootstrap.cc:654] T 789f118b81e04ad289cf4b4f9c9ecd85 P 659357c7635644ea8fad67cfbd3c3c22: Neither blocks nor log segments found. Creating new log.
I20260923 20:09:01.728895 4068 tablet_bootstrap.cc:654] T 789f118b81e04ad289cf4b4f9c9ecd85 P 98967392b7164818a63ed7e9912395c5: Neither blocks nor log segments found. Creating new log.
I20260923 20:09:01.731109 4069 tablet_bootstrap.cc:492] T 789f118b81e04ad289cf4b4f9c9ecd85 P 659357c7635644ea8fad67cfbd3c3c22: No bootstrap required, opened a new log
I20260923 20:09:01.731268 4069 ts_tablet_manager.cc:1397] T 789f118b81e04ad289cf4b4f9c9ecd85 P 659357c7635644ea8fad67cfbd3c3c22: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260923 20:09:01.731295 4067 tablet_bootstrap.cc:492] T 789f118b81e04ad289cf4b4f9c9ecd85 P df7bc6713ea546638a209a6b211dfa0c: No bootstrap required, opened a new log
I20260923 20:09:01.731412 4067 ts_tablet_manager.cc:1397] T 789f118b81e04ad289cf4b4f9c9ecd85 P df7bc6713ea546638a209a6b211dfa0c: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.000s
I20260923 20:09:01.732067 4069 raft_consensus.cc:348] T 789f118b81e04ad289cf4b4f9c9ecd85 P 659357c7635644ea8fad67cfbd3c3c22 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98967392b7164818a63ed7e9912395c5" member_type: VOTER last_known_addr { host: "127.1.170.129" port: 44623 } } peers { permanent_uuid: "df7bc6713ea546638a209a6b211dfa0c" member_type: VOTER last_known_addr { host: "127.1.170.130" port: 39109 } } peers { permanent_uuid: "659357c7635644ea8fad67cfbd3c3c22" member_type: VOTER last_known_addr { host: "127.1.170.131" port: 37945 } }
I20260923 20:09:01.732172 4067 raft_consensus.cc:348] T 789f118b81e04ad289cf4b4f9c9ecd85 P df7bc6713ea546638a209a6b211dfa0c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98967392b7164818a63ed7e9912395c5" member_type: VOTER last_known_addr { host: "127.1.170.129" port: 44623 } } peers { permanent_uuid: "df7bc6713ea546638a209a6b211dfa0c" member_type: VOTER last_known_addr { host: "127.1.170.130" port: 39109 } } peers { permanent_uuid: "659357c7635644ea8fad67cfbd3c3c22" member_type: VOTER last_known_addr { host: "127.1.170.131" port: 37945 } }
I20260923 20:09:01.732336 4069 raft_consensus.cc:374] T 789f118b81e04ad289cf4b4f9c9ecd85 P 659357c7635644ea8fad67cfbd3c3c22 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:09:01.732368 4067 raft_consensus.cc:374] T 789f118b81e04ad289cf4b4f9c9ecd85 P df7bc6713ea546638a209a6b211dfa0c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:09:01.732427 4069 raft_consensus.cc:729] T 789f118b81e04ad289cf4b4f9c9ecd85 P 659357c7635644ea8fad67cfbd3c3c22 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 659357c7635644ea8fad67cfbd3c3c22, State: Initialized, Role: FOLLOWER
I20260923 20:09:01.732466 4067 raft_consensus.cc:729] T 789f118b81e04ad289cf4b4f9c9ecd85 P df7bc6713ea546638a209a6b211dfa0c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df7bc6713ea546638a209a6b211dfa0c, State: Initialized, Role: FOLLOWER
I20260923 20:09:01.732687 4069 consensus_queue.cc:260] T 789f118b81e04ad289cf4b4f9c9ecd85 P 659357c7635644ea8fad67cfbd3c3c22 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98967392b7164818a63ed7e9912395c5" member_type: VOTER last_known_addr { host: "127.1.170.129" port: 44623 } } peers { permanent_uuid: "df7bc6713ea546638a209a6b211dfa0c" member_type: VOTER last_known_addr { host: "127.1.170.130" port: 39109 } } peers { permanent_uuid: "659357c7635644ea8fad67cfbd3c3c22" member_type: VOTER last_known_addr { host: "127.1.170.131" port: 37945 } }
I20260923 20:09:01.732687 4067 consensus_queue.cc:260] T 789f118b81e04ad289cf4b4f9c9ecd85 P df7bc6713ea546638a209a6b211dfa0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98967392b7164818a63ed7e9912395c5" member_type: VOTER last_known_addr { host: "127.1.170.129" port: 44623 } } peers { permanent_uuid: "df7bc6713ea546638a209a6b211dfa0c" member_type: VOTER last_known_addr { host: "127.1.170.130" port: 39109 } } peers { permanent_uuid: "659357c7635644ea8fad67cfbd3c3c22" member_type: VOTER last_known_addr { host: "127.1.170.131" port: 37945 } }
I20260923 20:09:01.733279 4069 ts_tablet_manager.cc:1428] T 789f118b81e04ad289cf4b4f9c9ecd85 P 659357c7635644ea8fad67cfbd3c3c22: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260923 20:09:01.734449 4068 tablet_bootstrap.cc:492] T 789f118b81e04ad289cf4b4f9c9ecd85 P 98967392b7164818a63ed7e9912395c5: No bootstrap required, opened a new log
I20260923 20:09:01.734651 4068 ts_tablet_manager.cc:1397] T 789f118b81e04ad289cf4b4f9c9ecd85 P 98967392b7164818a63ed7e9912395c5: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.000s
I20260923 20:09:01.735675 4068 raft_consensus.cc:348] T 789f118b81e04ad289cf4b4f9c9ecd85 P 98967392b7164818a63ed7e9912395c5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98967392b7164818a63ed7e9912395c5" member_type: VOTER last_known_addr { host: "127.1.170.129" port: 44623 } } peers { permanent_uuid: "df7bc6713ea546638a209a6b211dfa0c" member_type: VOTER last_known_addr { host: "127.1.170.130" port: 39109 } } peers { permanent_uuid: "659357c7635644ea8fad67cfbd3c3c22" member_type: VOTER last_known_addr { host: "127.1.170.131" port: 37945 } }
I20260923 20:09:01.736012 4068 raft_consensus.cc:374] T 789f118b81e04ad289cf4b4f9c9ecd85 P 98967392b7164818a63ed7e9912395c5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:09:01.736127 4068 raft_consensus.cc:729] T 789f118b81e04ad289cf4b4f9c9ecd85 P 98967392b7164818a63ed7e9912395c5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98967392b7164818a63ed7e9912395c5, State: Initialized, Role: FOLLOWER
I20260923 20:09:01.736451 4068 consensus_queue.cc:260] T 789f118b81e04ad289cf4b4f9c9ecd85 P 98967392b7164818a63ed7e9912395c5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98967392b7164818a63ed7e9912395c5" member_type: VOTER last_known_addr { host: "127.1.170.129" port: 44623 } } peers { permanent_uuid: "df7bc6713ea546638a209a6b211dfa0c" member_type: VOTER last_known_addr { host: "127.1.170.130" port: 39109 } } peers { permanent_uuid: "659357c7635644ea8fad67cfbd3c3c22" member_type: VOTER last_known_addr { host: "127.1.170.131" port: 37945 } }
I20260923 20:09:01.737208 4068 ts_tablet_manager.cc:1428] T 789f118b81e04ad289cf4b4f9c9ecd85 P 98967392b7164818a63ed7e9912395c5: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260923 20:09:01.737962 4067 ts_tablet_manager.cc:1428] T 789f118b81e04ad289cf4b4f9c9ecd85 P df7bc6713ea546638a209a6b211dfa0c: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260923 20:09:01.847738 4047 raft_consensus.cc:482] T 789f118b81e04ad289cf4b4f9c9ecd85 P 659357c7635644ea8fad67cfbd3c3c22 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 20:09:01.847932 4047 raft_consensus.cc:504] T 789f118b81e04ad289cf4b4f9c9ecd85 P 659357c7635644ea8fad67cfbd3c3c22 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98967392b7164818a63ed7e9912395c5" member_type: VOTER last_known_addr { host: "127.1.170.129" port: 44623 } } peers { permanent_uuid: "df7bc6713ea546638a209a6b211dfa0c" member_type: VOTER last_known_addr { host: "127.1.170.130" port: 39109 } } peers { permanent_uuid: "659357c7635644ea8fad67cfbd3c3c22" member_type: VOTER last_known_addr { host: "127.1.170.131" port: 37945 } }
I20260923 20:09:01.848497 4047 leader_election.cc:288] T 789f118b81e04ad289cf4b4f9c9ecd85 P 659357c7635644ea8fad67cfbd3c3c22 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 98967392b7164818a63ed7e9912395c5 (127.1.170.129:44623), df7bc6713ea546638a209a6b211dfa0c (127.1.170.130:39109)
I20260923 20:09:01.849164 3627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "789f118b81e04ad289cf4b4f9c9ecd85" candidate_uuid: "659357c7635644ea8fad67cfbd3c3c22" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98967392b7164818a63ed7e9912395c5" is_pre_election: true
I20260923 20:09:01.849443 3627 raft_consensus.cc:2454] T 789f118b81e04ad289cf4b4f9c9ecd85 P 98967392b7164818a63ed7e9912395c5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 659357c7635644ea8fad67cfbd3c3c22 in term 0.
I20260923 20:09:01.849570 3762 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "789f118b81e04ad289cf4b4f9c9ecd85" candidate_uuid: "659357c7635644ea8fad67cfbd3c3c22" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df7bc6713ea546638a209a6b211dfa0c" is_pre_election: true
I20260923 20:09:01.849792 3762 raft_consensus.cc:2454] T 789f118b81e04ad289cf4b4f9c9ecd85 P df7bc6713ea546638a209a6b211dfa0c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 659357c7635644ea8fad67cfbd3c3c22 in term 0.
I20260923 20:09:01.850003 3833 leader_election.cc:302] T 789f118b81e04ad289cf4b4f9c9ecd85 P 659357c7635644ea8fad67cfbd3c3c22 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 659357c7635644ea8fad67cfbd3c3c22, 98967392b7164818a63ed7e9912395c5; no voters:
I20260923 20:09:01.850404 4047 raft_consensus.cc:2781] T 789f118b81e04ad289cf4b4f9c9ecd85 P 659357c7635644ea8fad67cfbd3c3c22 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 20:09:01.850508 4047 raft_consensus.cc:482] T 789f118b81e04ad289cf4b4f9c9ecd85 P 659357c7635644ea8fad67cfbd3c3c22 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 20:09:01.850627 4047 raft_consensus.cc:3037] T 789f118b81e04ad289cf4b4f9c9ecd85 P 659357c7635644ea8fad67cfbd3c3c22 [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:09:01.852716 4047 raft_consensus.cc:504] T 789f118b81e04ad289cf4b4f9c9ecd85 P 659357c7635644ea8fad67cfbd3c3c22 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98967392b7164818a63ed7e9912395c5" member_type: VOTER last_known_addr { host: "127.1.170.129" port: 44623 } } peers { permanent_uuid: "df7bc6713ea546638a209a6b211dfa0c" member_type: VOTER last_known_addr { host: "127.1.170.130" port: 39109 } } peers { permanent_uuid: "659357c7635644ea8fad67cfbd3c3c22" member_type: VOTER last_known_addr { host: "127.1.170.131" port: 37945 } }
I20260923 20:09:01.853266 4047 leader_election.cc:288] T 789f118b81e04ad289cf4b4f9c9ecd85 P 659357c7635644ea8fad67cfbd3c3c22 [CANDIDATE]: Term 1 election: Requested vote from peers 98967392b7164818a63ed7e9912395c5 (127.1.170.129:44623), df7bc6713ea546638a209a6b211dfa0c (127.1.170.130:39109)
I20260923 20:09:01.853807 3627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "789f118b81e04ad289cf4b4f9c9ecd85" candidate_uuid: "659357c7635644ea8fad67cfbd3c3c22" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98967392b7164818a63ed7e9912395c5"
I20260923 20:09:01.854043 3627 raft_consensus.cc:3037] T 789f118b81e04ad289cf4b4f9c9ecd85 P 98967392b7164818a63ed7e9912395c5 [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:09:01.854050 3762 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "789f118b81e04ad289cf4b4f9c9ecd85" candidate_uuid: "659357c7635644ea8fad67cfbd3c3c22" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df7bc6713ea546638a209a6b211dfa0c"
I20260923 20:09:01.854307 3762 raft_consensus.cc:3037] T 789f118b81e04ad289cf4b4f9c9ecd85 P df7bc6713ea546638a209a6b211dfa0c [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:09:01.856163 3627 raft_consensus.cc:2454] T 789f118b81e04ad289cf4b4f9c9ecd85 P 98967392b7164818a63ed7e9912395c5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 659357c7635644ea8fad67cfbd3c3c22 in term 1.
I20260923 20:09:01.856276 3762 raft_consensus.cc:2454] T 789f118b81e04ad289cf4b4f9c9ecd85 P df7bc6713ea546638a209a6b211dfa0c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 659357c7635644ea8fad67cfbd3c3c22 in term 1.
I20260923 20:09:01.856554 3833 leader_election.cc:302] T 789f118b81e04ad289cf4b4f9c9ecd85 P 659357c7635644ea8fad67cfbd3c3c22 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 659357c7635644ea8fad67cfbd3c3c22, 98967392b7164818a63ed7e9912395c5; no voters:
I20260923 20:09:01.856822 4047 raft_consensus.cc:2781] T 789f118b81e04ad289cf4b4f9c9ecd85 P 659357c7635644ea8fad67cfbd3c3c22 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 20:09:01.856982 4047 raft_consensus.cc:686] T 789f118b81e04ad289cf4b4f9c9ecd85 P 659357c7635644ea8fad67cfbd3c3c22 [term 1 LEADER]: Becoming Leader. State: Replica: 659357c7635644ea8fad67cfbd3c3c22, State: Running, Role: LEADER
I20260923 20:09:01.857271 4047 consensus_queue.cc:237] T 789f118b81e04ad289cf4b4f9c9ecd85 P 659357c7635644ea8fad67cfbd3c3c22 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98967392b7164818a63ed7e9912395c5" member_type: VOTER last_known_addr { host: "127.1.170.129" port: 44623 } } peers { permanent_uuid: "df7bc6713ea546638a209a6b211dfa0c" member_type: VOTER last_known_addr { host: "127.1.170.130" port: 39109 } } peers { permanent_uuid: "659357c7635644ea8fad67cfbd3c3c22" member_type: VOTER last_known_addr { host: "127.1.170.131" port: 37945 } }
I20260923 20:09:01.860083 3465 catalog_manager.cc:5495] T 789f118b81e04ad289cf4b4f9c9ecd85 P 659357c7635644ea8fad67cfbd3c3c22 reported cstate change: term changed from 0 to 1, leader changed from <none> to 659357c7635644ea8fad67cfbd3c3c22 (127.1.170.131). New cstate: current_term: 1 leader_uuid: "659357c7635644ea8fad67cfbd3c3c22" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98967392b7164818a63ed7e9912395c5" member_type: VOTER last_known_addr { host: "127.1.170.129" port: 44623 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df7bc6713ea546638a209a6b211dfa0c" member_type: VOTER last_known_addr { host: "127.1.170.130" port: 39109 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "659357c7635644ea8fad67cfbd3c3c22" member_type: VOTER last_known_addr { host: "127.1.170.131" port: 37945 } health_report { overall_health: HEALTHY } } }
I20260923 20:09:01.895465 4076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:09:01.889604 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33468 (local address 127.1.170.131:37945)
0923 20:09:01.890243 (+ 639us) server_negotiation.cc:206] Beginning negotiation
0923 20:09:01.890248 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 20:09:01.890268 (+ 20us) server_negotiation.cc:407] Connection header received
0923 20:09:01.890334 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:09:01.890338 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:09:01.890402 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:09:01.890496 (+ 94us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 20:09:01.890885 (+ 389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:09:01.891809 (+ 924us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:09:01.893395 (+ 1586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:09:01.893846 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:09:01.894023 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:09:01.894548 (+ 525us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 20:09:01.894664 (+ 116us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 20:09:01.894751 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0923 20:09:01.895006 (+ 255us) server_negotiation.cc:299] Negotiation successful
0923 20:09:01.895085 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":518,"thread_start_us":375,"threads_started":1}
I20260923 20:09:01.904197 3762 raft_consensus.cc:1261] T 789f118b81e04ad289cf4b4f9c9ecd85 P df7bc6713ea546638a209a6b211dfa0c [term 1 FOLLOWER]: Refusing update from remote peer 659357c7635644ea8fad67cfbd3c3c22: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 20:09:01.904393 3627 raft_consensus.cc:1261] T 789f118b81e04ad289cf4b4f9c9ecd85 P 98967392b7164818a63ed7e9912395c5 [term 1 FOLLOWER]: Refusing update from remote peer 659357c7635644ea8fad67cfbd3c3c22: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 20:09:01.904878 4047 consensus_queue.cc:1035] T 789f118b81e04ad289cf4b4f9c9ecd85 P 659357c7635644ea8fad67cfbd3c3c22 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df7bc6713ea546638a209a6b211dfa0c" member_type: VOTER last_known_addr { host: "127.1.170.130" port: 39109 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 20:09:01.905107 4010 consensus_queue.cc:1035] T 789f118b81e04ad289cf4b4f9c9ecd85 P 659357c7635644ea8fad67cfbd3c3c22 [LEADER]: Connected to new peer: Peer: permanent_uuid: "98967392b7164818a63ed7e9912395c5" member_type: VOTER last_known_addr { host: "127.1.170.129" port: 44623 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 20:09:01.951980 3465 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34722:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 20:09:01.952209 3465 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34722:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 20:09:01.955580 3465 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4d74cac2e4f944208b16833320585dd4: Sending DeleteTablet for 3 replicas of tablet 789f118b81e04ad289cf4b4f9c9ecd85
I20260923 20:09:01.956511 3603 tablet_service.cc:1514] Processing DeleteTablet for tablet 789f118b81e04ad289cf4b4f9c9ecd85 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 20:09:01 UTC) from {username='kudu', principal='kudu/127.1.170.190@KRBTEST.COM'} at 127.0.0.1:38920
I20260923 20:09:01.956552 3882 tablet_service.cc:1514] Processing DeleteTablet for tablet 789f118b81e04ad289cf4b4f9c9ecd85 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 20:09:01 UTC) from {username='kudu', principal='kudu/127.1.170.190@KRBTEST.COM'} at 127.0.0.1:33456
I20260923 20:09:01.956559 3741 tablet_service.cc:1514] Processing DeleteTablet for tablet 789f118b81e04ad289cf4b4f9c9ecd85 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 20:09:01 UTC) from {username='kudu', principal='kudu/127.1.170.190@KRBTEST.COM'} at 127.0.0.1:38888
I20260923 20:09:01.956816 4056 tablet_replica.cc:330] T 789f118b81e04ad289cf4b4f9c9ecd85 P 98967392b7164818a63ed7e9912395c5: stopping tablet replica
I20260923 20:09:01.956820 4057 tablet_replica.cc:330] T 789f118b81e04ad289cf4b4f9c9ecd85 P 659357c7635644ea8fad67cfbd3c3c22: stopping tablet replica
I20260923 20:09:01.956858 4058 tablet_replica.cc:330] T 789f118b81e04ad289cf4b4f9c9ecd85 P df7bc6713ea546638a209a6b211dfa0c: stopping tablet replica
I20260923 20:09:01.957012 4056 raft_consensus.cc:2229] T 789f118b81e04ad289cf4b4f9c9ecd85 P 98967392b7164818a63ed7e9912395c5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 20:09:01.957015 4057 raft_consensus.cc:2229] T 789f118b81e04ad289cf4b4f9c9ecd85 P 659357c7635644ea8fad67cfbd3c3c22 [term 1 LEADER]: Raft consensus shutting down.
I20260923 20:09:01.957022 4058 raft_consensus.cc:2229] T 789f118b81e04ad289cf4b4f9c9ecd85 P df7bc6713ea546638a209a6b211dfa0c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 20:09:01.957213 4058 raft_consensus.cc:2258] T 789f118b81e04ad289cf4b4f9c9ecd85 P df7bc6713ea546638a209a6b211dfa0c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 20:09:01.957265 4056 raft_consensus.cc:2258] T 789f118b81e04ad289cf4b4f9c9ecd85 P 98967392b7164818a63ed7e9912395c5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 20:09:01.957440 4057 raft_consensus.cc:2258] T 789f118b81e04ad289cf4b4f9c9ecd85 P 659357c7635644ea8fad67cfbd3c3c22 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 20:09:01.958487 4058 ts_tablet_manager.cc:1905] T 789f118b81e04ad289cf4b4f9c9ecd85 P df7bc6713ea546638a209a6b211dfa0c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 20:09:01.958554 4056 ts_tablet_manager.cc:1905] T 789f118b81e04ad289cf4b4f9c9ecd85 P 98967392b7164818a63ed7e9912395c5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 20:09:01.958802 4057 ts_tablet_manager.cc:1905] T 789f118b81e04ad289cf4b4f9c9ecd85 P 659357c7635644ea8fad67cfbd3c3c22: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 20:09:01.963212 4057 ts_tablet_manager.cc:1918] T 789f118b81e04ad289cf4b4f9c9ecd85 P 659357c7635644ea8fad67cfbd3c3c22: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260923 20:09:01.963400 4057 log.cc:1196] T 789f118b81e04ad289cf4b4f9c9ecd85 P 659357c7635644ea8fad67cfbd3c3c22: Deleting WAL directory at /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790194121577027-1706-0/minicluster-data/ts-2/wal/wals/789f118b81e04ad289cf4b4f9c9ecd85
I20260923 20:09:01.964138 4057 ts_tablet_manager.cc:1939] T 789f118b81e04ad289cf4b4f9c9ecd85 P 659357c7635644ea8fad67cfbd3c3c22: Deleting consensus metadata
I20260923 20:09:01.964381 3465 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34722:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 20:09:01.965242 3465 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 20:09:01.965453 3453 catalog_manager.cc:4841] TS 659357c7635644ea8fad67cfbd3c3c22 (127.1.170.131:37945): tablet 789f118b81e04ad289cf4b4f9c9ecd85 (table test-table [id=b7cbd14a138b44fe8c860a3469b27bf8]) successfully deleted
I20260923 20:09:01.965500 4058 ts_tablet_manager.cc:1918] T 789f118b81e04ad289cf4b4f9c9ecd85 P df7bc6713ea546638a209a6b211dfa0c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260923 20:09:01.965500 4056 ts_tablet_manager.cc:1918] T 789f118b81e04ad289cf4b4f9c9ecd85 P 98967392b7164818a63ed7e9912395c5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260923 20:09:01.965653 4058 log.cc:1196] T 789f118b81e04ad289cf4b4f9c9ecd85 P df7bc6713ea546638a209a6b211dfa0c: Deleting WAL directory at /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790194121577027-1706-0/minicluster-data/ts-1/wal/wals/789f118b81e04ad289cf4b4f9c9ecd85
I20260923 20:09:01.965667 4056 log.cc:1196] T 789f118b81e04ad289cf4b4f9c9ecd85 P 98967392b7164818a63ed7e9912395c5: Deleting WAL directory at /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790194121577027-1706-0/minicluster-data/ts-0/wal/wals/789f118b81e04ad289cf4b4f9c9ecd85
I20260923 20:09:01.966140 4058 ts_tablet_manager.cc:1939] T 789f118b81e04ad289cf4b4f9c9ecd85 P df7bc6713ea546638a209a6b211dfa0c: Deleting consensus metadata
I20260923 20:09:01.966120 4056 ts_tablet_manager.cc:1939] T 789f118b81e04ad289cf4b4f9c9ecd85 P 98967392b7164818a63ed7e9912395c5: Deleting consensus metadata
I20260923 20:09:01.967209 3451 catalog_manager.cc:4841] TS 98967392b7164818a63ed7e9912395c5 (127.1.170.129:44623): tablet 789f118b81e04ad289cf4b4f9c9ecd85 (table test-table [id=b7cbd14a138b44fe8c860a3469b27bf8]) successfully deleted
I20260923 20:09:01.967451 3451 catalog_manager.cc:4841] TS df7bc6713ea546638a209a6b211dfa0c (127.1.170.130:39109): tablet 789f118b81e04ad289cf4b4f9c9ecd85 (table test-table [id=b7cbd14a138b44fe8c860a3469b27bf8]) successfully deleted
I20260923 20:09:01.972681 3603 tablet_service.cc:1467] Processing CreateTablet for tablet 49d94d0e5e104ee2b79b3859dad1d199 (DEFAULT_TABLE table=test-table [id=c989a8d644984a2f8c0cf06c88a79754]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 20:09:01.972705 3741 tablet_service.cc:1467] Processing CreateTablet for tablet 49d94d0e5e104ee2b79b3859dad1d199 (DEFAULT_TABLE table=test-table [id=c989a8d644984a2f8c0cf06c88a79754]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 20:09:01.972857 3882 tablet_service.cc:1467] Processing CreateTablet for tablet 49d94d0e5e104ee2b79b3859dad1d199 (DEFAULT_TABLE table=test-table [id=c989a8d644984a2f8c0cf06c88a79754]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 20:09:01.973127 3741 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 49d94d0e5e104ee2b79b3859dad1d199. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:09:01.973155 3603 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 49d94d0e5e104ee2b79b3859dad1d199. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:09:01.973276 3882 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 49d94d0e5e104ee2b79b3859dad1d199. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:09:01.978816 4069 tablet_bootstrap.cc:492] T 49d94d0e5e104ee2b79b3859dad1d199 P 659357c7635644ea8fad67cfbd3c3c22: Bootstrap starting.
I20260923 20:09:01.979473 4067 tablet_bootstrap.cc:492] T 49d94d0e5e104ee2b79b3859dad1d199 P df7bc6713ea546638a209a6b211dfa0c: Bootstrap starting.
I20260923 20:09:01.980499 4069 tablet_bootstrap.cc:654] T 49d94d0e5e104ee2b79b3859dad1d199 P 659357c7635644ea8fad67cfbd3c3c22: Neither blocks nor log segments found. Creating new log.
I20260923 20:09:01.981089 4067 tablet_bootstrap.cc:654] T 49d94d0e5e104ee2b79b3859dad1d199 P df7bc6713ea546638a209a6b211dfa0c: Neither blocks nor log segments found. Creating new log.
I20260923 20:09:01.981508 4068 tablet_bootstrap.cc:492] T 49d94d0e5e104ee2b79b3859dad1d199 P 98967392b7164818a63ed7e9912395c5: Bootstrap starting.
I20260923 20:09:01.983115 4068 tablet_bootstrap.cc:654] T 49d94d0e5e104ee2b79b3859dad1d199 P 98967392b7164818a63ed7e9912395c5: Neither blocks nor log segments found. Creating new log.
I20260923 20:09:01.983747 4067 tablet_bootstrap.cc:492] T 49d94d0e5e104ee2b79b3859dad1d199 P df7bc6713ea546638a209a6b211dfa0c: No bootstrap required, opened a new log
I20260923 20:09:01.983951 4067 ts_tablet_manager.cc:1397] T 49d94d0e5e104ee2b79b3859dad1d199 P df7bc6713ea546638a209a6b211dfa0c: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20260923 20:09:01.984714 4067 raft_consensus.cc:348] T 49d94d0e5e104ee2b79b3859dad1d199 P df7bc6713ea546638a209a6b211dfa0c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98967392b7164818a63ed7e9912395c5" member_type: VOTER last_known_addr { host: "127.1.170.129" port: 44623 } } peers { permanent_uuid: "df7bc6713ea546638a209a6b211dfa0c" member_type: VOTER last_known_addr { host: "127.1.170.130" port: 39109 } } peers { permanent_uuid: "659357c7635644ea8fad67cfbd3c3c22" member_type: VOTER last_known_addr { host: "127.1.170.131" port: 37945 } }
I20260923 20:09:01.985052 4067 raft_consensus.cc:374] T 49d94d0e5e104ee2b79b3859dad1d199 P df7bc6713ea546638a209a6b211dfa0c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:09:01.985196 4067 raft_consensus.cc:729] T 49d94d0e5e104ee2b79b3859dad1d199 P df7bc6713ea546638a209a6b211dfa0c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df7bc6713ea546638a209a6b211dfa0c, State: Initialized, Role: FOLLOWER
I20260923 20:09:01.985436 4069 tablet_bootstrap.cc:492] T 49d94d0e5e104ee2b79b3859dad1d199 P 659357c7635644ea8fad67cfbd3c3c22: No bootstrap required, opened a new log
I20260923 20:09:01.985601 4069 ts_tablet_manager.cc:1397] T 49d94d0e5e104ee2b79b3859dad1d199 P 659357c7635644ea8fad67cfbd3c3c22: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.000s
I20260923 20:09:01.985492 4067 consensus_queue.cc:260] T 49d94d0e5e104ee2b79b3859dad1d199 P df7bc6713ea546638a209a6b211dfa0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98967392b7164818a63ed7e9912395c5" member_type: VOTER last_known_addr { host: "127.1.170.129" port: 44623 } } peers { permanent_uuid: "df7bc6713ea546638a209a6b211dfa0c" member_type: VOTER last_known_addr { host: "127.1.170.130" port: 39109 } } peers { permanent_uuid: "659357c7635644ea8fad67cfbd3c3c22" member_type: VOTER last_known_addr { host: "127.1.170.131" port: 37945 } }
I20260923 20:09:01.986300 4069 raft_consensus.cc:348] T 49d94d0e5e104ee2b79b3859dad1d199 P 659357c7635644ea8fad67cfbd3c3c22 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98967392b7164818a63ed7e9912395c5" member_type: VOTER last_known_addr { host: "127.1.170.129" port: 44623 } } peers { permanent_uuid: "df7bc6713ea546638a209a6b211dfa0c" member_type: VOTER last_known_addr { host: "127.1.170.130" port: 39109 } } peers { permanent_uuid: "659357c7635644ea8fad67cfbd3c3c22" member_type: VOTER last_known_addr { host: "127.1.170.131" port: 37945 } }
I20260923 20:09:01.986589 4069 raft_consensus.cc:374] T 49d94d0e5e104ee2b79b3859dad1d199 P 659357c7635644ea8fad67cfbd3c3c22 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:09:01.986696 4067 ts_tablet_manager.cc:1428] T 49d94d0e5e104ee2b79b3859dad1d199 P df7bc6713ea546638a209a6b211dfa0c: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260923 20:09:01.986702 4069 raft_consensus.cc:729] T 49d94d0e5e104ee2b79b3859dad1d199 P 659357c7635644ea8fad67cfbd3c3c22 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 659357c7635644ea8fad67cfbd3c3c22, State: Initialized, Role: FOLLOWER
I20260923 20:09:01.986959 4069 consensus_queue.cc:260] T 49d94d0e5e104ee2b79b3859dad1d199 P 659357c7635644ea8fad67cfbd3c3c22 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98967392b7164818a63ed7e9912395c5" member_type: VOTER last_known_addr { host: "127.1.170.129" port: 44623 } } peers { permanent_uuid: "df7bc6713ea546638a209a6b211dfa0c" member_type: VOTER last_known_addr { host: "127.1.170.130" port: 39109 } } peers { permanent_uuid: "659357c7635644ea8fad67cfbd3c3c22" member_type: VOTER last_known_addr { host: "127.1.170.131" port: 37945 } }
I20260923 20:09:01.987419 4069 ts_tablet_manager.cc:1428] T 49d94d0e5e104ee2b79b3859dad1d199 P 659357c7635644ea8fad67cfbd3c3c22: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260923 20:09:01.988463 4068 tablet_bootstrap.cc:492] T 49d94d0e5e104ee2b79b3859dad1d199 P 98967392b7164818a63ed7e9912395c5: No bootstrap required, opened a new log
I20260923 20:09:01.988629 4068 ts_tablet_manager.cc:1397] T 49d94d0e5e104ee2b79b3859dad1d199 P 98967392b7164818a63ed7e9912395c5: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.000s
I20260923 20:09:01.989434 4068 raft_consensus.cc:348] T 49d94d0e5e104ee2b79b3859dad1d199 P 98967392b7164818a63ed7e9912395c5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98967392b7164818a63ed7e9912395c5" member_type: VOTER last_known_addr { host: "127.1.170.129" port: 44623 } } peers { permanent_uuid: "df7bc6713ea546638a209a6b211dfa0c" member_type: VOTER last_known_addr { host: "127.1.170.130" port: 39109 } } peers { permanent_uuid: "659357c7635644ea8fad67cfbd3c3c22" member_type: VOTER last_known_addr { host: "127.1.170.131" port: 37945 } }
I20260923 20:09:01.989667 4068 raft_consensus.cc:374] T 49d94d0e5e104ee2b79b3859dad1d199 P 98967392b7164818a63ed7e9912395c5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:09:01.989727 4068 raft_consensus.cc:729] T 49d94d0e5e104ee2b79b3859dad1d199 P 98967392b7164818a63ed7e9912395c5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98967392b7164818a63ed7e9912395c5, State: Initialized, Role: FOLLOWER
I20260923 20:09:01.990159 4068 consensus_queue.cc:260] T 49d94d0e5e104ee2b79b3859dad1d199 P 98967392b7164818a63ed7e9912395c5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98967392b7164818a63ed7e9912395c5" member_type: VOTER last_known_addr { host: "127.1.170.129" port: 44623 } } peers { permanent_uuid: "df7bc6713ea546638a209a6b211dfa0c" member_type: VOTER last_known_addr { host: "127.1.170.130" port: 39109 } } peers { permanent_uuid: "659357c7635644ea8fad67cfbd3c3c22" member_type: VOTER last_known_addr { host: "127.1.170.131" port: 37945 } }
I20260923 20:09:01.990396 4082 raft_consensus.cc:482] T 49d94d0e5e104ee2b79b3859dad1d199 P 98967392b7164818a63ed7e9912395c5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 20:09:01.990514 4082 raft_consensus.cc:504] T 49d94d0e5e104ee2b79b3859dad1d199 P 98967392b7164818a63ed7e9912395c5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98967392b7164818a63ed7e9912395c5" member_type: VOTER last_known_addr { host: "127.1.170.129" port: 44623 } } peers { permanent_uuid: "df7bc6713ea546638a209a6b211dfa0c" member_type: VOTER last_known_addr { host: "127.1.170.130" port: 39109 } } peers { permanent_uuid: "659357c7635644ea8fad67cfbd3c3c22" member_type: VOTER last_known_addr { host: "127.1.170.131" port: 37945 } }
I20260923 20:09:01.991120 4082 leader_election.cc:288] T 49d94d0e5e104ee2b79b3859dad1d199 P 98967392b7164818a63ed7e9912395c5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df7bc6713ea546638a209a6b211dfa0c (127.1.170.130:39109), 659357c7635644ea8fad67cfbd3c3c22 (127.1.170.131:37945)
I20260923 20:09:01.991585 4068 ts_tablet_manager.cc:1428] T 49d94d0e5e104ee2b79b3859dad1d199 P 98967392b7164818a63ed7e9912395c5: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260923 20:09:01.991708 3901 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49d94d0e5e104ee2b79b3859dad1d199" candidate_uuid: "98967392b7164818a63ed7e9912395c5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "659357c7635644ea8fad67cfbd3c3c22" is_pre_election: true
I20260923 20:09:01.991940 3901 raft_consensus.cc:2454] T 49d94d0e5e104ee2b79b3859dad1d199 P 659357c7635644ea8fad67cfbd3c3c22 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 98967392b7164818a63ed7e9912395c5 in term 0.
I20260923 20:09:01.992534 3556 leader_election.cc:302] T 49d94d0e5e104ee2b79b3859dad1d199 P 98967392b7164818a63ed7e9912395c5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 659357c7635644ea8fad67cfbd3c3c22, 98967392b7164818a63ed7e9912395c5; no voters:
I20260923 20:09:01.992414 3762 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49d94d0e5e104ee2b79b3859dad1d199" candidate_uuid: "98967392b7164818a63ed7e9912395c5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df7bc6713ea546638a209a6b211dfa0c" is_pre_election: true
I20260923 20:09:01.992708 3762 raft_consensus.cc:2454] T 49d94d0e5e104ee2b79b3859dad1d199 P df7bc6713ea546638a209a6b211dfa0c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 98967392b7164818a63ed7e9912395c5 in term 0.
I20260923 20:09:01.992785 4082 raft_consensus.cc:2781] T 49d94d0e5e104ee2b79b3859dad1d199 P 98967392b7164818a63ed7e9912395c5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 20:09:01.992895 4082 raft_consensus.cc:482] T 49d94d0e5e104ee2b79b3859dad1d199 P 98967392b7164818a63ed7e9912395c5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 20:09:01.992977 4082 raft_consensus.cc:3037] T 49d94d0e5e104ee2b79b3859dad1d199 P 98967392b7164818a63ed7e9912395c5 [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:09:01.996472 4082 raft_consensus.cc:504] T 49d94d0e5e104ee2b79b3859dad1d199 P 98967392b7164818a63ed7e9912395c5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98967392b7164818a63ed7e9912395c5" member_type: VOTER last_known_addr { host: "127.1.170.129" port: 44623 } } peers { permanent_uuid: "df7bc6713ea546638a209a6b211dfa0c" member_type: VOTER last_known_addr { host: "127.1.170.130" port: 39109 } } peers { permanent_uuid: "659357c7635644ea8fad67cfbd3c3c22" member_type: VOTER last_known_addr { host: "127.1.170.131" port: 37945 } }
I20260923 20:09:01.997761 4082 leader_election.cc:288] T 49d94d0e5e104ee2b79b3859dad1d199 P 98967392b7164818a63ed7e9912395c5 [CANDIDATE]: Term 1 election: Requested vote from peers df7bc6713ea546638a209a6b211dfa0c (127.1.170.130:39109), 659357c7635644ea8fad67cfbd3c3c22 (127.1.170.131:37945)
I20260923 20:09:01.998519 3901 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49d94d0e5e104ee2b79b3859dad1d199" candidate_uuid: "98967392b7164818a63ed7e9912395c5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "659357c7635644ea8fad67cfbd3c3c22"
I20260923 20:09:01.998780 3901 raft_consensus.cc:3037] T 49d94d0e5e104ee2b79b3859dad1d199 P 659357c7635644ea8fad67cfbd3c3c22 [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:09:01.999238 3762 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49d94d0e5e104ee2b79b3859dad1d199" candidate_uuid: "98967392b7164818a63ed7e9912395c5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df7bc6713ea546638a209a6b211dfa0c"
I20260923 20:09:01.999447 3762 raft_consensus.cc:3037] T 49d94d0e5e104ee2b79b3859dad1d199 P df7bc6713ea546638a209a6b211dfa0c [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:09:02.002385 3762 raft_consensus.cc:2454] T 49d94d0e5e104ee2b79b3859dad1d199 P df7bc6713ea546638a209a6b211dfa0c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 98967392b7164818a63ed7e9912395c5 in term 1.
I20260923 20:09:02.002385 3901 raft_consensus.cc:2454] T 49d94d0e5e104ee2b79b3859dad1d199 P 659357c7635644ea8fad67cfbd3c3c22 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 98967392b7164818a63ed7e9912395c5 in term 1.
I20260923 20:09:02.002938 3554 leader_election.cc:302] T 49d94d0e5e104ee2b79b3859dad1d199 P 98967392b7164818a63ed7e9912395c5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 98967392b7164818a63ed7e9912395c5, df7bc6713ea546638a209a6b211dfa0c; no voters:
I20260923 20:09:02.003473 4082 raft_consensus.cc:2781] T 49d94d0e5e104ee2b79b3859dad1d199 P 98967392b7164818a63ed7e9912395c5 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 20:09:02.003645 4082 raft_consensus.cc:686] T 49d94d0e5e104ee2b79b3859dad1d199 P 98967392b7164818a63ed7e9912395c5 [term 1 LEADER]: Becoming Leader. State: Replica: 98967392b7164818a63ed7e9912395c5, State: Running, Role: LEADER
I20260923 20:09:02.003942 4082 consensus_queue.cc:237] T 49d94d0e5e104ee2b79b3859dad1d199 P 98967392b7164818a63ed7e9912395c5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98967392b7164818a63ed7e9912395c5" member_type: VOTER last_known_addr { host: "127.1.170.129" port: 44623 } } peers { permanent_uuid: "df7bc6713ea546638a209a6b211dfa0c" member_type: VOTER last_known_addr { host: "127.1.170.130" port: 39109 } } peers { permanent_uuid: "659357c7635644ea8fad67cfbd3c3c22" member_type: VOTER last_known_addr { host: "127.1.170.131" port: 37945 } }
I20260923 20:09:02.007242 3466 catalog_manager.cc:5495] T 49d94d0e5e104ee2b79b3859dad1d199 P 98967392b7164818a63ed7e9912395c5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 98967392b7164818a63ed7e9912395c5 (127.1.170.129). New cstate: current_term: 1 leader_uuid: "98967392b7164818a63ed7e9912395c5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98967392b7164818a63ed7e9912395c5" member_type: VOTER last_known_addr { host: "127.1.170.129" port: 44623 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "df7bc6713ea546638a209a6b211dfa0c" member_type: VOTER last_known_addr { host: "127.1.170.130" port: 39109 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "659357c7635644ea8fad67cfbd3c3c22" member_type: VOTER last_known_addr { host: "127.1.170.131" port: 37945 } health_report { overall_health: UNKNOWN } } }
I20260923 20:09:02.037441 4087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:09:02.031775 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38940 (local address 127.1.170.129:44623)
0923 20:09:02.032343 (+ 568us) server_negotiation.cc:206] Beginning negotiation
0923 20:09:02.032348 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 20:09:02.032367 (+ 19us) server_negotiation.cc:407] Connection header received
0923 20:09:02.032419 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:09:02.032422 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:09:02.032477 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:09:02.032565 (+ 88us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 20:09:02.032984 (+ 419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:09:02.033812 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:09:02.035562 (+ 1750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:09:02.036012 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:09:02.036117 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:09:02.036607 (+ 490us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 20:09:02.036712 (+ 105us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 20:09:02.036791 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0923 20:09:02.036980 (+ 189us) server_negotiation.cc:299] Negotiation successful
0923 20:09:02.037093 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":443,"thread_start_us":295,"threads_started":1}
I20260923 20:09:02.052822 4087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:09:02.046805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.170.129:59445 (local address 127.1.170.129:44623)
0923 20:09:02.047031 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0923 20:09:02.047039 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 20:09:02.047692 (+ 653us) server_negotiation.cc:407] Connection header received
0923 20:09:02.047806 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:09:02.047811 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:09:02.047911 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:09:02.048036 (+ 125us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 20:09:02.048652 (+ 616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:09:02.049629 (+ 977us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:09:02.051169 (+ 1540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:09:02.051697 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:09:02.051801 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:09:02.052168 (+ 367us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 20:09:02.052280 (+ 112us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 20:09:02.052354 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0923 20:09:02.052514 (+ 160us) server_negotiation.cc:299] Negotiation successful
0923 20:09:02.052573 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260923 20:09:02.052977 4090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:09:02.046907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.129:44623 (local address 127.1.170.129:59445)
0923 20:09:02.047511 (+ 604us) negotiation.cc:107] Waiting for socket to connect
0923 20:09:02.047526 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0923 20:09:02.047644 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:09:02.048098 (+ 454us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:09:02.048102 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:09:02.048113 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0923 20:09:02.048426 (+ 313us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:09:02.048435 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:09:02.049789 (+ 1354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:09:02.049793 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:09:02.051035 (+ 1242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:09:02.051047 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:09:02.051856 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:09:02.051861 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:09:02.051972 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:09:02.052012 (+ 40us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0923 20:09:02.052394 (+ 382us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0923 20:09:02.052404 (+ 10us) client_negotiation.cc:769] Sending connection context
0923 20:09:02.052480 (+ 76us) client_negotiation.cc:240] Negotiation successful
0923 20:09:02.052601 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":473,"thread_start_us":355,"threads_started":1}
I20260923 20:09:02.060884 3901 raft_consensus.cc:1261] T 49d94d0e5e104ee2b79b3859dad1d199 P 659357c7635644ea8fad67cfbd3c3c22 [term 1 FOLLOWER]: Refusing update from remote peer 98967392b7164818a63ed7e9912395c5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 20:09:02.061415 3762 raft_consensus.cc:1261] T 49d94d0e5e104ee2b79b3859dad1d199 P df7bc6713ea546638a209a6b211dfa0c [term 1 FOLLOWER]: Refusing update from remote peer 98967392b7164818a63ed7e9912395c5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 20:09:02.061458 4051 consensus_queue.cc:1035] T 49d94d0e5e104ee2b79b3859dad1d199 P 98967392b7164818a63ed7e9912395c5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "659357c7635644ea8fad67cfbd3c3c22" member_type: VOTER last_known_addr { host: "127.1.170.131" port: 37945 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 20:09:02.062196 4051 consensus_queue.cc:1035] T 49d94d0e5e104ee2b79b3859dad1d199 P 98967392b7164818a63ed7e9912395c5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df7bc6713ea546638a209a6b211dfa0c" member_type: VOTER last_known_addr { host: "127.1.170.130" port: 39109 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260923 20:09:02.081007 4036 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260923 20:09:03.685508 3466 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34722:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 20:09:03.686246 3466 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34722:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 20:09:03.692106 3466 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4d74cac2e4f944208b16833320585dd4: Sending DeleteTablet for 3 replicas of tablet 49d94d0e5e104ee2b79b3859dad1d199
I20260923 20:09:03.693046 3602 tablet_service.cc:1514] Processing DeleteTablet for tablet 49d94d0e5e104ee2b79b3859dad1d199 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 20:09:03 UTC) from {username='kudu', principal='kudu/127.1.170.190@KRBTEST.COM'} at 127.0.0.1:38920
I20260923 20:09:03.693086 3882 tablet_service.cc:1514] Processing DeleteTablet for tablet 49d94d0e5e104ee2b79b3859dad1d199 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 20:09:03 UTC) from {username='kudu', principal='kudu/127.1.170.190@KRBTEST.COM'} at 127.0.0.1:33456
I20260923 20:09:03.693141 3741 tablet_service.cc:1514] Processing DeleteTablet for tablet 49d94d0e5e104ee2b79b3859dad1d199 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 20:09:03 UTC) from {username='kudu', principal='kudu/127.1.170.190@KRBTEST.COM'} at 127.0.0.1:38888
I20260923 20:09:03.694010 1706 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskqUuku6/build/asan/bin/kudu pid 3537
I20260923 20:09:03.704316 4113 tablet_replica.cc:330] T 49d94d0e5e104ee2b79b3859dad1d199 P 98967392b7164818a63ed7e9912395c5: stopping tablet replica
I20260923 20:09:03.704936 4113 raft_consensus.cc:2229] T 49d94d0e5e104ee2b79b3859dad1d199 P 98967392b7164818a63ed7e9912395c5 [term 1 LEADER]: Raft consensus shutting down.
I20260923 20:09:03.706050 4114 tablet_replica.cc:330] T 49d94d0e5e104ee2b79b3859dad1d199 P 659357c7635644ea8fad67cfbd3c3c22: stopping tablet replica
I20260923 20:09:03.706202 4113 raft_consensus.cc:2258] T 49d94d0e5e104ee2b79b3859dad1d199 P 98967392b7164818a63ed7e9912395c5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 20:09:03.706548 4114 raft_consensus.cc:2229] T 49d94d0e5e104ee2b79b3859dad1d199 P 659357c7635644ea8fad67cfbd3c3c22 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 20:09:03.706930 4114 raft_consensus.cc:2258] T 49d94d0e5e104ee2b79b3859dad1d199 P 659357c7635644ea8fad67cfbd3c3c22 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 20:09:03.707610 4113 ts_tablet_manager.cc:1905] T 49d94d0e5e104ee2b79b3859dad1d199 P 98967392b7164818a63ed7e9912395c5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 20:09:03.707736 4115 tablet_replica.cc:330] T 49d94d0e5e104ee2b79b3859dad1d199 P df7bc6713ea546638a209a6b211dfa0c: stopping tablet replica
I20260923 20:09:03.707985 4115 raft_consensus.cc:2229] T 49d94d0e5e104ee2b79b3859dad1d199 P df7bc6713ea546638a209a6b211dfa0c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 20:09:03.708330 4115 raft_consensus.cc:2258] T 49d94d0e5e104ee2b79b3859dad1d199 P df7bc6713ea546638a209a6b211dfa0c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 20:09:03.708974 4114 ts_tablet_manager.cc:1905] T 49d94d0e5e104ee2b79b3859dad1d199 P 659357c7635644ea8fad67cfbd3c3c22: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 20:09:03.709518 4115 ts_tablet_manager.cc:1905] T 49d94d0e5e104ee2b79b3859dad1d199 P df7bc6713ea546638a209a6b211dfa0c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 20:09:03.716732 4113 ts_tablet_manager.cc:1918] T 49d94d0e5e104ee2b79b3859dad1d199 P 98967392b7164818a63ed7e9912395c5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260923 20:09:03.717005 4113 log.cc:1196] T 49d94d0e5e104ee2b79b3859dad1d199 P 98967392b7164818a63ed7e9912395c5: Deleting WAL directory at /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790194121577027-1706-0/minicluster-data/ts-0/wal/wals/49d94d0e5e104ee2b79b3859dad1d199
I20260923 20:09:03.717628 4113 ts_tablet_manager.cc:1939] T 49d94d0e5e104ee2b79b3859dad1d199 P 98967392b7164818a63ed7e9912395c5: Deleting consensus metadata
I20260923 20:09:03.717810 4114 ts_tablet_manager.cc:1918] T 49d94d0e5e104ee2b79b3859dad1d199 P 659357c7635644ea8fad67cfbd3c3c22: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260923 20:09:03.717799 4115 ts_tablet_manager.cc:1918] T 49d94d0e5e104ee2b79b3859dad1d199 P df7bc6713ea546638a209a6b211dfa0c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260923 20:09:03.718019 4115 log.cc:1196] T 49d94d0e5e104ee2b79b3859dad1d199 P df7bc6713ea546638a209a6b211dfa0c: Deleting WAL directory at /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790194121577027-1706-0/minicluster-data/ts-1/wal/wals/49d94d0e5e104ee2b79b3859dad1d199
I20260923 20:09:03.718027 4114 log.cc:1196] T 49d94d0e5e104ee2b79b3859dad1d199 P 659357c7635644ea8fad67cfbd3c3c22: Deleting WAL directory at /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790194121577027-1706-0/minicluster-data/ts-2/wal/wals/49d94d0e5e104ee2b79b3859dad1d199
I20260923 20:09:03.718503 4114 ts_tablet_manager.cc:1939] T 49d94d0e5e104ee2b79b3859dad1d199 P 659357c7635644ea8fad67cfbd3c3c22: Deleting consensus metadata
I20260923 20:09:03.718552 4115 ts_tablet_manager.cc:1939] T 49d94d0e5e104ee2b79b3859dad1d199 P df7bc6713ea546638a209a6b211dfa0c: Deleting consensus metadata
I20260923 20:09:03.719139 3451 catalog_manager.cc:4841] TS 98967392b7164818a63ed7e9912395c5 (127.1.170.129:44623): tablet 49d94d0e5e104ee2b79b3859dad1d199 (table test-table [id=c989a8d644984a2f8c0cf06c88a79754]) successfully deleted
I20260923 20:09:03.719672 3453 catalog_manager.cc:4841] TS 659357c7635644ea8fad67cfbd3c3c22 (127.1.170.131:37945): tablet 49d94d0e5e104ee2b79b3859dad1d199 (table test-table [id=c989a8d644984a2f8c0cf06c88a79754]) successfully deleted
I20260923 20:09:03.719738 3451 catalog_manager.cc:4841] TS df7bc6713ea546638a209a6b211dfa0c (127.1.170.130:39109): tablet 49d94d0e5e104ee2b79b3859dad1d199 (table test-table [id=c989a8d644984a2f8c0cf06c88a79754]) successfully deleted
Sep 23 20:09:03 dist-test-slave-th4g krb5kdc[3421](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194141, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.1.170.129@KRBTEST.COM
I20260923 20:09:03.742302 4117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:09:03.695599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52512 (local address 127.1.170.129:44623)
0923 20:09:03.710547 (+ 14948us) server_negotiation.cc:206] Beginning negotiation
0923 20:09:03.710555 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 20:09:03.711905 (+ 1350us) server_negotiation.cc:407] Connection header received
0923 20:09:03.711979 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:09:03.711985 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:09:03.712073 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:09:03.712169 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:09:03.715065 (+ 2896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:09:03.716630 (+ 1565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:09:03.718807 (+ 2177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:09:03.719410 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:09:03.719543 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:09:03.735721 (+ 16178us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:09:03.735739 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:09:03.735757 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:09:03.735801 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:09:03.739004 (+ 3203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:09:03.739855 (+ 851us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:09:03.739863 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:09:03.739865 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:09:03.740023 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:09:03.740501 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:09:03.740505 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:09:03.740508 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:09:03.740966 (+ 458us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:09:03.741347 (+ 381us) server_negotiation.cc:1035] Waiting for connection context
0923 20:09:03.741607 (+ 260us) server_negotiation.cc:299] Negotiation successful
0923 20:09:03.741830 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14771,"thread_start_us":21903,"threads_started":1}
W20260923 20:09:03.743330 3668 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52512
W20260923 20:09:03.744426 1706 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskqUuku6/build/asan/bin/kudu pid 3537: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260923 20:09:03.744840 1706 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqUuku6/build/asan/bin/kudu with pid 3537
I20260923 20:09:03.781283 1706 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskqUuku6/build/asan/bin/kudu pid 3677
Sep 23 20:09:03 dist-test-slave-th4g krb5kdc[3421](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194141, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.1.170.130@KRBTEST.COM
I20260923 20:09:03.800834 4118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:09:03.782149 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33984 (local address 127.1.170.130:39109)
0923 20:09:03.782671 (+ 522us) server_negotiation.cc:206] Beginning negotiation
0923 20:09:03.782676 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 20:09:03.782693 (+ 17us) server_negotiation.cc:407] Connection header received
0923 20:09:03.782759 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:09:03.782764 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:09:03.782822 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:09:03.782931 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:09:03.784313 (+ 1382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:09:03.785089 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:09:03.786568 (+ 1479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:09:03.786987 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:09:03.787101 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:09:03.792842 (+ 5741us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:09:03.792858 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:09:03.792875 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:09:03.792924 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:09:03.796944 (+ 4020us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:09:03.797934 (+ 990us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:09:03.797943 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:09:03.797945 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:09:03.798268 (+ 323us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:09:03.798715 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:09:03.798722 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:09:03.798725 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:09:03.799224 (+ 499us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:09:03.799592 (+ 368us) server_negotiation.cc:1035] Waiting for connection context
0923 20:09:03.799968 (+ 376us) server_negotiation.cc:299] Negotiation successful
0923 20:09:03.800232 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":417,"thread_start_us":271,"threads_started":1}
W20260923 20:09:03.802493 3807 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33984
W20260923 20:09:03.803488 1706 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskqUuku6/build/asan/bin/kudu pid 3677: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260923 20:09:03.803702 1706 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqUuku6/build/asan/bin/kudu with pid 3677
I20260923 20:09:03.822822 1706 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskqUuku6/build/asan/bin/kudu pid 3816
Sep 23 20:09:03 dist-test-slave-th4g krb5kdc[3421](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194141, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.1.170.131@KRBTEST.COM
I20260923 20:09:03.841295 4119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:09:03.823954 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55932 (local address 127.1.170.131:37945)
0923 20:09:03.824647 (+ 693us) server_negotiation.cc:206] Beginning negotiation
0923 20:09:03.824653 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:09:03.824672 (+ 19us) server_negotiation.cc:407] Connection header received
0923 20:09:03.824731 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:09:03.824735 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:09:03.824823 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:09:03.824922 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:09:03.826749 (+ 1827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:09:03.827491 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:09:03.828800 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:09:03.829364 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:09:03.829485 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:09:03.835062 (+ 5577us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:09:03.835078 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:09:03.835095 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:09:03.835139 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:09:03.838257 (+ 3118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:09:03.839060 (+ 803us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:09:03.839075 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:09:03.839077 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:09:03.839218 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:09:03.839619 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:09:03.839624 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:09:03.839629 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:09:03.840004 (+ 375us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:09:03.840324 (+ 320us) server_negotiation.cc:1035] Waiting for connection context
0923 20:09:03.840615 (+ 291us) server_negotiation.cc:299] Negotiation successful
0923 20:09:03.840880 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":429,"thread_start_us":277,"threads_started":1}
W20260923 20:09:03.842151 3947 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55932
W20260923 20:09:03.842928 1706 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskqUuku6/build/asan/bin/kudu pid 3816: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260923 20:09:03.843097 1706 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqUuku6/build/asan/bin/kudu with pid 3816
I20260923 20:09:03.868939 1706 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskqUuku6/build/asan/bin/kudu pid 3434
I20260923 20:09:03.870715 3506 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:09:04.001161 1706 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqUuku6/build/asan/bin/kudu with pid 3434
2026-09-23T20:09:04Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (8743 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790194121577027-1706-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 20:09:04 dist-test-slave-th4g krb5kdc[4123](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 20:09:04 dist-test-slave-th4g krb5kdc[4123](info): set up 2 sockets
Sep 23 20:09:04 dist-test-slave-th4g krb5kdc[4123](info): commencing operation
krb5kdc: starting...
W20260923 20:09:05.174531 1706 mini_kdc.cc:121] Time spent starting KDC: real 1.117s user 0.001s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790194121577027-1706-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790194121577027-1706-0/krb5kdc/test-user.keytab.
Sep 23 20:09:05 dist-test-slave-th4g krb5kdc[4123](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194145, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T20:09:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T20:09:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.170.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.170.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.170.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790194121577027-1706-0/krb5kdc/kudu_127.1.170.190.keytab.
Entry for principal kudu/127.1.170.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790194121577027-1706-0/krb5kdc/kudu_127.1.170.190.keytab.
I20260923 20:09:05.383667 1706 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqUuku6/build/asan/bin/kudu
/tmp/dist-test-taskqUuku6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790194121577027-1706-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790194121577027-1706-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790194121577027-1706-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790194121577027-1706-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.170.190:35971
--webserver_interface=127.1.170.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790194121577027-1706-0/krb5kdc/kudu_127.1.170.190.keytab
--principal=kudu/127.1.170.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.170.148:43765
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.170.190:35971
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790194121577027-1706-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790194121577027-1706-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790194121577027-1706-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:09:05.727180 4136 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:09:05.727531 4136 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:09:05.727699 4136 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:09:05.735773 4136 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 20:09:05.735935 4136 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:09:05.736042 4136 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:09:05.736140 4136 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 20:09:05.736228 4136 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 20:09:05.746572 4136 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.170.148:43765
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790194121577027-1706-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790194121577027-1706-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.170.190:35971
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790194121577027-1706-0/krb5kdc/kudu_127.1.170.190.keytab
--principal=kudu/127.1.170.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.170.190:35971
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790194121577027-1706-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.170.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790194121577027-1706-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:09:05.748441 4136 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:09:05.749959 4136 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:09:05.766052 4141 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:09:05.766026 4142 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:09:05.766345 4144 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:09:05.768496 4136 server_base.cc:1034] running on GCE node
I20260923 20:09:05.769356 4136 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:09:05.771358 4136 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:09:05.772619 4136 hybrid_clock.cc:648] HybridClock initialized: now 1790194145772558 us; error 56 us; skew 500 ppm
I20260923 20:09:05.773021 4136 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:09:05 dist-test-slave-th4g krb5kdc[4123](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194145, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.170.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:09:05.776408 4136 init.cc:377] Logged in from keytab as kudu/127.1.170.190@KRBTEST.COM (short username kudu)
I20260923 20:09:05.778090 4136 webserver.cc:460] Webserver started at http://127.1.170.190:41699/ using document root <none> and password file <none>
I20260923 20:09:05.778810 4136 fs_manager.cc:359] Metadata directory not provided
I20260923 20:09:05.778939 4136 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:09:05.779295 4136 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:09:05.782555 4136 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790194121577027-1706-0/minicluster-data/master-0/data/instance:
uuid: "05d0856af92d4b6690febe0f34f68e08"
format_stamp: "Formatted at 2026-09-23 20:09:05 on dist-test-slave-th4g"
server_key: "f6c87c27b480be2bfe152914b157a57e"
server_key_iv: "feadd46134f72da4d927d7a599ce8d97"
server_key_version: "encryptionkey@0"
I20260923 20:09:05.783398 4136 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790194121577027-1706-0/minicluster-data/master-0/wal/instance:
uuid: "05d0856af92d4b6690febe0f34f68e08"
format_stamp: "Formatted at 2026-09-23 20:09:05 on dist-test-slave-th4g"
server_key: "f6c87c27b480be2bfe152914b157a57e"
server_key_iv: "feadd46134f72da4d927d7a599ce8d97"
server_key_version: "encryptionkey@0"
I20260923 20:09:05.788910 4136 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260923 20:09:05.792960 4151 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:09:05.794660 4136 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260923 20:09:05.794879 4136 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790194121577027-1706-0/minicluster-data/master-0/data,/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790194121577027-1706-0/minicluster-data/master-0/wal
uuid: "05d0856af92d4b6690febe0f34f68e08"
format_stamp: "Formatted at 2026-09-23 20:09:05 on dist-test-slave-th4g"
server_key: "f6c87c27b480be2bfe152914b157a57e"
server_key_iv: "feadd46134f72da4d927d7a599ce8d97"
server_key_version: "encryptionkey@0"
I20260923 20:09:05.795097 4136 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790194121577027-1706-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790194121577027-1706-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790194121577027-1706-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:09:05.825346 4136 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:09:05.829864 4136 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:09:05.830190 4136 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:09:05.850348 4136 rpc_server.cc:307] RPC server started. Bound to: 127.1.170.190:35971
I20260923 20:09:05.850365 4202 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.170.190:35971 every 8 connection(s)
I20260923 20:09:05.851837 4136 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790194121577027-1706-0/minicluster-data/master-0/data/info.pb
I20260923 20:09:05.855248 1706 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqUuku6/build/asan/bin/kudu as pid 4136
I20260923 20:09:05.855548 1706 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790194121577027-1706-0/minicluster-data/master-0/wal/instance
I20260923 20:09:05.856348 1706 external_mini_cluster.cc:1428] Setting key dce2560d9eaa9401d43f033e9b7d8f54
I20260923 20:09:05.856529 4203 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:09:05.870278 4203 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 05d0856af92d4b6690febe0f34f68e08: Bootstrap starting.
Sep 23 20:09:05 dist-test-slave-th4g krb5kdc[4123](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194145, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.170.190@KRBTEST.COM
I20260923 20:09:05.873952 4203 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 05d0856af92d4b6690febe0f34f68e08: Neither blocks nor log segments found. Creating new log.
I20260923 20:09:05.875272 4203 log.cc:824] T 00000000000000000000000000000000 P 05d0856af92d4b6690febe0f34f68e08: Log is configured to *not* fsync() on all Append() calls
I20260923 20:09:05.878278 4203 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 05d0856af92d4b6690febe0f34f68e08: No bootstrap required, opened a new log
I20260923 20:09:05.881021 4206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:09:05.858897 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46560 (local address 127.1.170.190:35971)
0923 20:09:05.859953 (+ 1056us) server_negotiation.cc:206] Beginning negotiation
0923 20:09:05.859970 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0923 20:09:05.860042 (+ 72us) server_negotiation.cc:407] Connection header received
0923 20:09:05.862316 (+ 2274us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:09:05.862349 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:09:05.862849 (+ 500us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:09:05.863388 (+ 539us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:09:05.864901 (+ 1513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:09:05.866244 (+ 1343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:09:05.867539 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:09:05.868266 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:09:05.868481 (+ 215us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:09:05.872769 (+ 4288us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:09:05.872778 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:09:05.872794 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:09:05.872837 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:09:05.876494 (+ 3657us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:09:05.877290 (+ 796us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:09:05.877296 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:09:05.877302 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:09:05.877462 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:09:05.877848 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:09:05.877855 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:09:05.877858 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:09:05.878472 (+ 614us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:09:05.878787 (+ 315us) server_negotiation.cc:1035] Waiting for connection context
0923 20:09:05.879247 (+ 460us) server_negotiation.cc:299] Negotiation successful
0923 20:09:05.879685 (+ 438us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":762,"thread_start_us":481,"threads_started":1}
I20260923 20:09:05.887342 4203 raft_consensus.cc:348] T 00000000000000000000000000000000 P 05d0856af92d4b6690febe0f34f68e08 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05d0856af92d4b6690febe0f34f68e08" member_type: VOTER last_known_addr { host: "127.1.170.190" port: 35971 } }
I20260923 20:09:05.887784 4203 raft_consensus.cc:374] T 00000000000000000000000000000000 P 05d0856af92d4b6690febe0f34f68e08 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:09:05.887894 4203 raft_consensus.cc:729] T 00000000000000000000000000000000 P 05d0856af92d4b6690febe0f34f68e08 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 05d0856af92d4b6690febe0f34f68e08, State: Initialized, Role: FOLLOWER
I20260923 20:09:05.888442 4203 consensus_queue.cc:260] T 00000000000000000000000000000000 P 05d0856af92d4b6690febe0f34f68e08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05d0856af92d4b6690febe0f34f68e08" member_type: VOTER last_known_addr { host: "127.1.170.190" port: 35971 } }
I20260923 20:09:05.888723 4203 raft_consensus.cc:388] T 00000000000000000000000000000000 P 05d0856af92d4b6690febe0f34f68e08 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 20:09:05.888852 4203 raft_consensus.cc:482] T 00000000000000000000000000000000 P 05d0856af92d4b6690febe0f34f68e08 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 20:09:05.889035 4203 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 05d0856af92d4b6690febe0f34f68e08 [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:09:05.891438 4203 raft_consensus.cc:504] T 00000000000000000000000000000000 P 05d0856af92d4b6690febe0f34f68e08 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05d0856af92d4b6690febe0f34f68e08" member_type: VOTER last_known_addr { host: "127.1.170.190" port: 35971 } }
I20260923 20:09:05.891959 4203 leader_election.cc:302] T 00000000000000000000000000000000 P 05d0856af92d4b6690febe0f34f68e08 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 05d0856af92d4b6690febe0f34f68e08; no voters:
I20260923 20:09:05.892756 4203 leader_election.cc:288] T 00000000000000000000000000000000 P 05d0856af92d4b6690febe0f34f68e08 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 20:09:05.892886 4208 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 05d0856af92d4b6690febe0f34f68e08 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 20:09:05.894064 4208 raft_consensus.cc:686] T 00000000000000000000000000000000 P 05d0856af92d4b6690febe0f34f68e08 [term 1 LEADER]: Becoming Leader. State: Replica: 05d0856af92d4b6690febe0f34f68e08, State: Running, Role: LEADER
I20260923 20:09:05.894619 4208 consensus_queue.cc:237] T 00000000000000000000000000000000 P 05d0856af92d4b6690febe0f34f68e08 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05d0856af92d4b6690febe0f34f68e08" member_type: VOTER last_known_addr { host: "127.1.170.190" port: 35971 } }
I20260923 20:09:05.895190 4203 sys_catalog.cc:564] T 00000000000000000000000000000000 P 05d0856af92d4b6690febe0f34f68e08 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 20:09:05.897429 4209 sys_catalog.cc:455] T 00000000000000000000000000000000 P 05d0856af92d4b6690febe0f34f68e08 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "05d0856af92d4b6690febe0f34f68e08" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05d0856af92d4b6690febe0f34f68e08" member_type: VOTER last_known_addr { host: "127.1.170.190" port: 35971 } } }
I20260923 20:09:05.898031 4209 sys_catalog.cc:458] T 00000000000000000000000000000000 P 05d0856af92d4b6690febe0f34f68e08 [sys.catalog]: This master's current role is: LEADER
I20260923 20:09:05.897924 4210 sys_catalog.cc:455] T 00000000000000000000000000000000 P 05d0856af92d4b6690febe0f34f68e08 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 05d0856af92d4b6690febe0f34f68e08. Latest consensus state: current_term: 1 leader_uuid: "05d0856af92d4b6690febe0f34f68e08" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05d0856af92d4b6690febe0f34f68e08" member_type: VOTER last_known_addr { host: "127.1.170.190" port: 35971 } } }
I20260923 20:09:05.898207 4210 sys_catalog.cc:458] T 00000000000000000000000000000000 P 05d0856af92d4b6690febe0f34f68e08 [sys.catalog]: This master's current role is: LEADER
I20260923 20:09:05.900823 4217 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 20:09:05.908636 4217 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 20:09:05.919525 4217 catalog_manager.cc:1347] Generated new cluster ID: 3fb2bfe2ee314010a21102c16903aa4f
I20260923 20:09:05.919683 4217 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 20:09:05.938943 4217 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 20:09:05.940423 4217 catalog_manager.cc:1504] Loading token signing keys...
I20260923 20:09:05.947144 4217 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 05d0856af92d4b6690febe0f34f68e08: Generated new TSK 0
I20260923 20:09:05.948029 4217 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.170.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.170.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.170.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790194121577027-1706-0/krb5kdc/kudu_127.1.170.129.keytab.
Entry for principal kudu/127.1.170.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790194121577027-1706-0/krb5kdc/kudu_127.1.170.129.keytab.
I20260923 20:09:06.011807 1706 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqUuku6/build/asan/bin/kudu
/tmp/dist-test-taskqUuku6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790194121577027-1706-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790194121577027-1706-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790194121577027-1706-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790194121577027-1706-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.170.129:0
--local_ip_for_outbound_sockets=127.1.170.129
--webserver_interface=127.1.170.129
--webserver_port=0
--tserver_master_addrs=127.1.170.190:35971
--keytab_file=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790194121577027-1706-0/krb5kdc/kudu_127.1.170.129.keytab
--principal=kudu/127.1.170.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.170.148:43765
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790194121577027-1706-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790194121577027-1706-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790194121577027-1706-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:09:06.376240 4229 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:09:06.376529 4229 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:09:06.376698 4229 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:09:06.384389 4229 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:09:06.384549 4229 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:09:06.384733 4229 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.170.129
I20260923 20:09:06.395260 4229 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.170.148:43765
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790194121577027-1706-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790194121577027-1706-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790194121577027-1706-0/krb5kdc/kudu_127.1.170.129.keytab
--principal=kudu/127.1.170.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.170.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790194121577027-1706-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.170.129
--webserver_port=0
--tserver_master_addrs=127.1.170.190:35971
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.170.129
--log_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790194121577027-1706-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:09:06.396787 4229 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:09:06.398319 4229 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:09:06.411670 4234 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:09:06.411731 4235 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:09:06.411963 4237 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:09:06.412633 4229 server_base.cc:1034] running on GCE node
I20260923 20:09:06.413327 4229 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:09:06.414408 4229 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:09:06.415638 4229 hybrid_clock.cc:648] HybridClock initialized: now 1790194146415576 us; error 67 us; skew 500 ppm
I20260923 20:09:06.415992 4229 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:09:06 dist-test-slave-th4g krb5kdc[4123](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194146, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.170.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:09:06.419529 4229 init.cc:377] Logged in from keytab as kudu/127.1.170.129@KRBTEST.COM (short username kudu)
I20260923 20:09:06.421276 4229 webserver.cc:460] Webserver started at http://127.1.170.129:41341/ using document root <none> and password file <none>
I20260923 20:09:06.422094 4229 fs_manager.cc:359] Metadata directory not provided
I20260923 20:09:06.422243 4229 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:09:06.422580 4229 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:09:06.425722 4229 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790194121577027-1706-0/minicluster-data/ts-0/data/instance:
uuid: "4b3cdfb4e4b844848e6e33a440c7de36"
format_stamp: "Formatted at 2026-09-23 20:09:06 on dist-test-slave-th4g"
server_key: "c9fa9e202964e743f1b85c117e486ecd"
server_key_iv: "fce0b8427af2ea6312ba784ffcec7b41"
server_key_version: "encryptionkey@0"
I20260923 20:09:06.426721 4229 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790194121577027-1706-0/minicluster-data/ts-0/wal/instance:
uuid: "4b3cdfb4e4b844848e6e33a440c7de36"
format_stamp: "Formatted at 2026-09-23 20:09:06 on dist-test-slave-th4g"
server_key: "c9fa9e202964e743f1b85c117e486ecd"
server_key_iv: "fce0b8427af2ea6312ba784ffcec7b41"
server_key_version: "encryptionkey@0"
I20260923 20:09:06.432732 4229 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260923 20:09:06.437327 4244 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:09:06.438865 4229 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 20:09:06.439045 4229 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790194121577027-1706-0/minicluster-data/ts-0/data,/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790194121577027-1706-0/minicluster-data/ts-0/wal
uuid: "4b3cdfb4e4b844848e6e33a440c7de36"
format_stamp: "Formatted at 2026-09-23 20:09:06 on dist-test-slave-th4g"
server_key: "c9fa9e202964e743f1b85c117e486ecd"
server_key_iv: "fce0b8427af2ea6312ba784ffcec7b41"
server_key_version: "encryptionkey@0"
I20260923 20:09:06.439265 4229 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790194121577027-1706-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790194121577027-1706-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790194121577027-1706-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:09:06.456863 4229 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:09:06.461457 4229 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:09:06.461779 4229 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:09:06.462908 4229 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 20:09:06.464493 4229 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 20:09:06.464567 4229 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 20:09:06.464699 4229 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 20:09:06.464766 4229 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 20:09:06.500583 4229 rpc_server.cc:307] RPC server started. Bound to: 127.1.170.129:33649
I20260923 20:09:06.500615 4356 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.170.129:33649 every 8 connection(s)
I20260923 20:09:06.501947 4229 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790194121577027-1706-0/minicluster-data/ts-0/data/info.pb
I20260923 20:09:06.505267 1706 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqUuku6/build/asan/bin/kudu as pid 4229
I20260923 20:09:06.505493 1706 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790194121577027-1706-0/minicluster-data/ts-0/wal/instance
I20260923 20:09:06.506301 1706 external_mini_cluster.cc:1428] Setting key e3d0b40a034ecd69db92763b546244e7
Sep 23 20:09:06 dist-test-slave-th4g krb5kdc[4123](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194146, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.170.129@KRBTEST.COM for kudu/127.1.170.190@KRBTEST.COM
I20260923 20:09:06.525552 4359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:09:06.504470 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.170.129:41871 (local address 127.1.170.190:35971)
0923 20:09:06.505009 (+ 539us) server_negotiation.cc:206] Beginning negotiation
0923 20:09:06.505016 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:09:06.506126 (+ 1110us) server_negotiation.cc:407] Connection header received
0923 20:09:06.508501 (+ 2375us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:09:06.508506 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:09:06.508584 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:09:06.508710 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:09:06.510361 (+ 1651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:09:06.511520 (+ 1159us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:09:06.513085 (+ 1565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:09:06.513660 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:09:06.513808 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:09:06.518154 (+ 4346us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:09:06.518160 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:09:06.518163 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:09:06.518213 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:09:06.521345 (+ 3132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:09:06.522926 (+ 1581us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:09:06.522931 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:09:06.522933 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:09:06.523062 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:09:06.523620 (+ 558us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:09:06.523624 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:09:06.523627 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:09:06.523915 (+ 288us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:09:06.524042 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0923 20:09:06.525054 (+ 1012us) server_negotiation.cc:299] Negotiation successful
0923 20:09:06.525288 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":388,"thread_start_us":262,"threads_started":1}
WARNING: no policy specified for kudu/127.1.170.130@KRBTEST.COM; defaulting to no policy
I20260923 20:09:06.526727 4360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:09:06.505061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.190:35971 (local address 127.1.170.129:41871)
0923 20:09:06.505937 (+ 876us) negotiation.cc:107] Waiting for socket to connect
0923 20:09:06.505998 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0923 20:09:06.508169 (+ 2171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:09:06.508983 (+ 814us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:09:06.508991 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:09:06.509518 (+ 527us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:09:06.510183 (+ 665us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:09:06.510209 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:09:06.511688 (+ 1479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:09:06.511692 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:09:06.512939 (+ 1247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:09:06.512951 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:09:06.513940 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:09:06.513948 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:09:06.514189 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:09:06.514996 (+ 807us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:09:06.515022 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:09:06.517763 (+ 2741us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:09:06.521638 (+ 3875us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:09:06.521651 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:09:06.521677 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:09:06.522748 (+ 1071us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:09:06.523190 (+ 442us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:09:06.523195 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:09:06.523199 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:09:06.523488 (+ 289us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:09:06.524067 (+ 579us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:09:06.524076 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:09:06.524584 (+ 508us) client_negotiation.cc:769] Sending connection context
0923 20:09:06.524923 (+ 339us) client_negotiation.cc:240] Negotiation successful
0923 20:09:06.525533 (+ 610us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":538,"thread_start_us":355,"threads_started":1}
I20260923 20:09:06.528461 4357 heartbeater.cc:347] Connected to a master server at 127.1.170.190:35971
I20260923 20:09:06.528928 4357 heartbeater.cc:464] Registering TS with master...
I20260923 20:09:06.529673 4357 heartbeater.cc:511] Master 127.1.170.190:35971 requested a full tablet report, sending...
I20260923 20:09:06.532051 4168 ts_manager.cc:194] Registered new tserver with Master: 4b3cdfb4e4b844848e6e33a440c7de36 (127.1.170.129:33649)
I20260923 20:09:06.533761 4168 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.170.129@KRBTEST.COM'} at 127.1.170.129:41871
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.170.130@KRBTEST.COM" created.
I20260923 20:09:07.537480 4357 heartbeater.cc:503] Master 127.1.170.190:35971 was elected leader, sending a full tablet report...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.170.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790194121577027-1706-0/krb5kdc/kudu_127.1.170.130.keytab.
Entry for principal kudu/127.1.170.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790194121577027-1706-0/krb5kdc/kudu_127.1.170.130.keytab.
W20260923 20:09:07.606894 1706 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.1.170.130: real 1.098s user 0.002s sys 0.008s
I20260923 20:09:07.608299 1706 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqUuku6/build/asan/bin/kudu
/tmp/dist-test-taskqUuku6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790194121577027-1706-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790194121577027-1706-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790194121577027-1706-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790194121577027-1706-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.170.130:0
--local_ip_for_outbound_sockets=127.1.170.130
--webserver_interface=127.1.170.130
--webserver_port=0
--tserver_master_addrs=127.1.170.190:35971
--keytab_file=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790194121577027-1706-0/krb5kdc/kudu_127.1.170.130.keytab
--principal=kudu/127.1.170.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.170.148:43765
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790194121577027-1706-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790194121577027-1706-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790194121577027-1706-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:09:08.071949 4363 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:09:08.072280 4363 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:09:08.072378 4363 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:09:08.080552 4363 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:09:08.080698 4363 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:09:08.080895 4363 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.170.130
I20260923 20:09:08.092669 4363 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.170.148:43765
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790194121577027-1706-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790194121577027-1706-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790194121577027-1706-0/krb5kdc/kudu_127.1.170.130.keytab
--principal=kudu/127.1.170.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.170.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790194121577027-1706-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.170.130
--webserver_port=0
--tserver_master_addrs=127.1.170.190:35971
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.170.130
--log_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790194121577027-1706-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:09:08.094213 4363 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:09:08.095700 4363 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:09:08.116289 4368 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:09:08.116255 4369 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:09:08.116434 4371 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:09:08.117607 4363 server_base.cc:1034] running on GCE node
I20260923 20:09:08.118655 4363 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:09:08.119792 4363 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:09:08.121104 4363 hybrid_clock.cc:648] HybridClock initialized: now 1790194148121019 us; error 53 us; skew 500 ppm
I20260923 20:09:08.121778 4363 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:09:08 dist-test-slave-th4g krb5kdc[4123](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194148, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.170.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:09:08.126786 4363 init.cc:377] Logged in from keytab as kudu/127.1.170.130@KRBTEST.COM (short username kudu)
I20260923 20:09:08.128907 4363 webserver.cc:460] Webserver started at http://127.1.170.130:36341/ using document root <none> and password file <none>
I20260923 20:09:08.130254 4363 fs_manager.cc:359] Metadata directory not provided
I20260923 20:09:08.130467 4363 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:09:08.130831 4363 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:09:08.135133 4363 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790194121577027-1706-0/minicluster-data/ts-1/data/instance:
uuid: "d0939363ee8047b2b56a3050b57c2529"
format_stamp: "Formatted at 2026-09-23 20:09:08 on dist-test-slave-th4g"
server_key: "9cce5b37245fc4677039368c40b98579"
server_key_iv: "cfc1c007b8526b485df21cf08c22d52b"
server_key_version: "encryptionkey@0"
I20260923 20:09:08.136231 4363 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790194121577027-1706-0/minicluster-data/ts-1/wal/instance:
uuid: "d0939363ee8047b2b56a3050b57c2529"
format_stamp: "Formatted at 2026-09-23 20:09:08 on dist-test-slave-th4g"
server_key: "9cce5b37245fc4677039368c40b98579"
server_key_iv: "cfc1c007b8526b485df21cf08c22d52b"
server_key_version: "encryptionkey@0"
I20260923 20:09:08.144570 4363 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.001s sys 0.008s
I20260923 20:09:08.150966 4378 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:09:08.153280 4363 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.004s sys 0.000s
I20260923 20:09:08.153555 4363 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790194121577027-1706-0/minicluster-data/ts-1/data,/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790194121577027-1706-0/minicluster-data/ts-1/wal
uuid: "d0939363ee8047b2b56a3050b57c2529"
format_stamp: "Formatted at 2026-09-23 20:09:08 on dist-test-slave-th4g"
server_key: "9cce5b37245fc4677039368c40b98579"
server_key_iv: "cfc1c007b8526b485df21cf08c22d52b"
server_key_version: "encryptionkey@0"
I20260923 20:09:08.153807 4363 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790194121577027-1706-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790194121577027-1706-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790194121577027-1706-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:09:08.173461 4363 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:09:08.178946 4363 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:09:08.179215 4363 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:09:08.180212 4363 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 20:09:08.182785 4363 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 20:09:08.182920 4363 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 20:09:08.183059 4363 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 20:09:08.183138 4363 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 20:09:08.230412 4363 rpc_server.cc:307] RPC server started. Bound to: 127.1.170.130:44111
I20260923 20:09:08.230468 4490 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.170.130:44111 every 8 connection(s)
I20260923 20:09:08.231853 4363 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790194121577027-1706-0/minicluster-data/ts-1/data/info.pb
I20260923 20:09:08.236403 1706 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqUuku6/build/asan/bin/kudu as pid 4363
I20260923 20:09:08.236657 1706 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790194121577027-1706-0/minicluster-data/ts-1/wal/instance
I20260923 20:09:08.237210 1706 external_mini_cluster.cc:1428] Setting key b6e4711d0e75ee4d5a131ca66a93af53
Sep 23 20:09:08 dist-test-slave-th4g krb5kdc[4123](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194148, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.170.130@KRBTEST.COM for kudu/127.1.170.190@KRBTEST.COM
I20260923 20:09:08.259488 4493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:09:08.235021 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.170.130:38865 (local address 127.1.170.190:35971)
0923 20:09:08.235521 (+ 500us) server_negotiation.cc:206] Beginning negotiation
0923 20:09:08.235526 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 20:09:08.237212 (+ 1686us) server_negotiation.cc:407] Connection header received
0923 20:09:08.239555 (+ 2343us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:09:08.239560 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:09:08.239648 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:09:08.239781 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:09:08.241975 (+ 2194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:09:08.242770 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:09:08.244585 (+ 1815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:09:08.245020 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:09:08.245167 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:09:08.251325 (+ 6158us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:09:08.251331 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:09:08.251335 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:09:08.251385 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:09:08.254867 (+ 3482us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:09:08.256107 (+ 1240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:09:08.256115 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:09:08.256121 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:09:08.256328 (+ 207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:09:08.256858 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:09:08.256865 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:09:08.256870 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:09:08.257328 (+ 458us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:09:08.257508 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0923 20:09:08.258696 (+ 1188us) server_negotiation.cc:299] Negotiation successful
0923 20:09:08.259015 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":386,"thread_start_us":241,"threads_started":1}
I20260923 20:09:08.260349 4494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:09:08.235591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.190:35971 (local address 127.1.170.130:38865)
0923 20:09:08.237030 (+ 1439us) negotiation.cc:107] Waiting for socket to connect
0923 20:09:08.237073 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0923 20:09:08.239253 (+ 2180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:09:08.240071 (+ 818us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:09:08.240081 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:09:08.240953 (+ 872us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:09:08.241754 (+ 801us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:09:08.241776 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:09:08.242959 (+ 1183us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:09:08.242964 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:09:08.244431 (+ 1467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:09:08.244444 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:09:08.245447 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:09:08.245459 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:09:08.245777 (+ 318us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:09:08.246660 (+ 883us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:09:08.246689 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:09:08.250896 (+ 4207us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:09:08.255090 (+ 4194us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:09:08.255103 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:09:08.255230 (+ 127us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:09:08.255874 (+ 644us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:09:08.256498 (+ 624us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:09:08.256502 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:09:08.256504 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:09:08.256709 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:09:08.257710 (+ 1001us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:09:08.257719 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:09:08.258224 (+ 505us) client_negotiation.cc:769] Sending connection context
0923 20:09:08.258491 (+ 267us) client_negotiation.cc:240] Negotiation successful
0923 20:09:08.259024 (+ 533us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":933,"thread_start_us":524,"threads_started":1}
WARNING: no policy specified for kudu/127.1.170.131@KRBTEST.COM; defaulting to no policy
I20260923 20:09:08.264451 4491 heartbeater.cc:347] Connected to a master server at 127.1.170.190:35971
I20260923 20:09:08.265453 4491 heartbeater.cc:464] Registering TS with master...
I20260923 20:09:08.266820 4491 heartbeater.cc:511] Master 127.1.170.190:35971 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.170.131@KRBTEST.COM" created.
I20260923 20:09:08.269240 4168 ts_manager.cc:194] Registered new tserver with Master: d0939363ee8047b2b56a3050b57c2529 (127.1.170.130:44111)
I20260923 20:09:08.270166 4168 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.170.130@KRBTEST.COM'} at 127.1.170.130:38865
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.170.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790194121577027-1706-0/krb5kdc/kudu_127.1.170.131.keytab.
Entry for principal kudu/127.1.170.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790194121577027-1706-0/krb5kdc/kudu_127.1.170.131.keytab.
I20260923 20:09:08.307897 1706 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqUuku6/build/asan/bin/kudu
/tmp/dist-test-taskqUuku6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790194121577027-1706-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790194121577027-1706-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790194121577027-1706-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790194121577027-1706-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.170.131:0
--local_ip_for_outbound_sockets=127.1.170.131
--webserver_interface=127.1.170.131
--webserver_port=0
--tserver_master_addrs=127.1.170.190:35971
--keytab_file=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790194121577027-1706-0/krb5kdc/kudu_127.1.170.131.keytab
--principal=kudu/127.1.170.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.170.148:43765
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790194121577027-1706-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790194121577027-1706-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790194121577027-1706-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:09:08.749588 4497 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:09:08.749993 4497 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:09:08.750097 4497 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:09:08.758468 4497 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:09:08.758591 4497 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:09:08.758716 4497 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.170.131
I20260923 20:09:08.771708 4497 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.170.148:43765
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790194121577027-1706-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790194121577027-1706-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790194121577027-1706-0/krb5kdc/kudu_127.1.170.131.keytab
--principal=kudu/127.1.170.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.170.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790194121577027-1706-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.170.131
--webserver_port=0
--tserver_master_addrs=127.1.170.190:35971
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.170.131
--log_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790194121577027-1706-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:09:08.773289 4497 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:09:08.774945 4497 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:09:08.792300 4503 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:09:08.792167 4502 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:09:08.792261 4505 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:09:08.793301 4497 server_base.cc:1034] running on GCE node
I20260923 20:09:08.794301 4497 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:09:08.795512 4497 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:09:08.796752 4497 hybrid_clock.cc:648] HybridClock initialized: now 1790194148796687 us; error 70 us; skew 500 ppm
I20260923 20:09:08.797149 4497 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:09:08 dist-test-slave-th4g krb5kdc[4123](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194148, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.170.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:09:08.810192 4497 init.cc:377] Logged in from keytab as kudu/127.1.170.131@KRBTEST.COM (short username kudu)
I20260923 20:09:08.812129 4497 webserver.cc:460] Webserver started at http://127.1.170.131:37611/ using document root <none> and password file <none>
I20260923 20:09:08.812947 4497 fs_manager.cc:359] Metadata directory not provided
I20260923 20:09:08.813058 4497 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:09:08.813350 4497 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:09:08.816581 4497 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790194121577027-1706-0/minicluster-data/ts-2/data/instance:
uuid: "74bd6b8c86ce44d0b2787426bc4af704"
format_stamp: "Formatted at 2026-09-23 20:09:08 on dist-test-slave-th4g"
server_key: "59111862c0cf6c36d1ffe2499da61c92"
server_key_iv: "7a6a3e154630a5519649064021590b83"
server_key_version: "encryptionkey@0"
I20260923 20:09:08.818188 4497 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790194121577027-1706-0/minicluster-data/ts-2/wal/instance:
uuid: "74bd6b8c86ce44d0b2787426bc4af704"
format_stamp: "Formatted at 2026-09-23 20:09:08 on dist-test-slave-th4g"
server_key: "59111862c0cf6c36d1ffe2499da61c92"
server_key_iv: "7a6a3e154630a5519649064021590b83"
server_key_version: "encryptionkey@0"
I20260923 20:09:08.826104 4497 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.003s sys 0.004s
I20260923 20:09:08.830694 4512 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:09:08.832985 4497 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260923 20:09:08.833312 4497 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790194121577027-1706-0/minicluster-data/ts-2/data,/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790194121577027-1706-0/minicluster-data/ts-2/wal
uuid: "74bd6b8c86ce44d0b2787426bc4af704"
format_stamp: "Formatted at 2026-09-23 20:09:08 on dist-test-slave-th4g"
server_key: "59111862c0cf6c36d1ffe2499da61c92"
server_key_iv: "7a6a3e154630a5519649064021590b83"
server_key_version: "encryptionkey@0"
I20260923 20:09:08.833537 4497 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790194121577027-1706-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790194121577027-1706-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790194121577027-1706-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:09:08.847404 4497 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:09:08.854192 4497 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:09:08.854769 4497 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:09:08.856328 4497 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 20:09:08.858088 4497 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 20:09:08.858175 4497 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 20:09:08.858304 4497 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 20:09:08.858371 4497 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 20:09:08.901537 4497 rpc_server.cc:307] RPC server started. Bound to: 127.1.170.131:35531
I20260923 20:09:08.901573 4624 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.170.131:35531 every 8 connection(s)
I20260923 20:09:08.903175 4497 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790194121577027-1706-0/minicluster-data/ts-2/data/info.pb
I20260923 20:09:08.907859 1706 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqUuku6/build/asan/bin/kudu as pid 4497
I20260923 20:09:08.908437 1706 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790194121577027-1706-0/minicluster-data/ts-2/wal/instance
I20260923 20:09:08.909555 1706 external_mini_cluster.cc:1428] Setting key 733b3248eae5461cfbd5c863b78c36b8
Sep 23 20:09:08 dist-test-slave-th4g krb5kdc[4123](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194148, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.170.131@KRBTEST.COM for kudu/127.1.170.190@KRBTEST.COM
I20260923 20:09:08.933526 4627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:09:08.906814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.170.131:39583 (local address 127.1.170.190:35971)
0923 20:09:08.907721 (+ 907us) server_negotiation.cc:206] Beginning negotiation
0923 20:09:08.907729 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 20:09:08.909125 (+ 1396us) server_negotiation.cc:407] Connection header received
0923 20:09:08.912417 (+ 3292us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:09:08.912424 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:09:08.912522 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:09:08.912648 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:09:08.915013 (+ 2365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:09:08.915802 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:09:08.918626 (+ 2824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:09:08.919118 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:09:08.919352 (+ 234us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:09:08.924476 (+ 5124us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:09:08.924482 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:09:08.924485 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:09:08.924527 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:09:08.929117 (+ 4590us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:09:08.930624 (+ 1507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:09:08.930630 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:09:08.930633 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:09:08.930830 (+ 197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:09:08.931455 (+ 625us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:09:08.931460 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:09:08.931462 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:09:08.931849 (+ 387us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:09:08.931995 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0923 20:09:08.932981 (+ 986us) server_negotiation.cc:299] Negotiation successful
0923 20:09:08.933219 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":639,"thread_start_us":441,"threads_started":1}
I20260923 20:09:08.934788 4628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:09:08.907368 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.190:35971 (local address 127.1.170.131:39583)
0923 20:09:08.908836 (+ 1468us) negotiation.cc:107] Waiting for socket to connect
0923 20:09:08.908889 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0923 20:09:08.912018 (+ 3129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:09:08.913098 (+ 1080us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:09:08.913159 (+ 61us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:09:08.913926 (+ 767us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:09:08.914821 (+ 895us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:09:08.914844 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:09:08.916277 (+ 1433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:09:08.916283 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:09:08.918199 (+ 1916us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:09:08.918238 (+ 39us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:09:08.919511 (+ 1273us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:09:08.919522 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:09:08.919818 (+ 296us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:09:08.920862 (+ 1044us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:09:08.920943 (+ 81us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:09:08.924111 (+ 3168us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:09:08.929399 (+ 5288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:09:08.929412 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:09:08.929439 (+ 27us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:09:08.930388 (+ 949us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:09:08.931025 (+ 637us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:09:08.931030 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:09:08.931033 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:09:08.931276 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:09:08.932056 (+ 780us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:09:08.932063 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:09:08.932533 (+ 470us) client_negotiation.cc:769] Sending connection context
0923 20:09:08.932828 (+ 295us) client_negotiation.cc:240] Negotiation successful
0923 20:09:08.933312 (+ 484us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1137,"thread_start_us":910,"threads_started":1}
I20260923 20:09:08.936823 4625 heartbeater.cc:347] Connected to a master server at 127.1.170.190:35971
I20260923 20:09:08.937270 4625 heartbeater.cc:464] Registering TS with master...
I20260923 20:09:08.938414 4625 heartbeater.cc:511] Master 127.1.170.190:35971 requested a full tablet report, sending...
I20260923 20:09:08.940883 4168 ts_manager.cc:194] Registered new tserver with Master: 74bd6b8c86ce44d0b2787426bc4af704 (127.1.170.131:35531)
I20260923 20:09:08.942965 4168 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.170.131@KRBTEST.COM'} at 127.1.170.131:39583
I20260923 20:09:08.943835 1706 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260923 20:09:08.980890 4627 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 20:09:08.978024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46570 (local address 127.1.170.190:35971)
0923 20:09:08.978346 (+ 322us) server_negotiation.cc:206] Beginning negotiation
0923 20:09:08.978351 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 20:09:08.979050 (+ 699us) server_negotiation.cc:407] Connection header received
0923 20:09:08.979119 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:09:08.979123 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:09:08.979223 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:09:08.979328 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:09:08.980671 (+ 1343us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46570: BlockingRecv error: recv got EOF from 127.0.0.1:46570 (error 108)
Metrics: {"server-negotiator.queue_time_us":95}
I20260923 20:09:08.985980 1706 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskqUuku6/build/asan/bin/kudu pid 4229
W20260923 20:09:08.989616 1706 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskqUuku6/build/asan/bin/kudu pid 4229: Not authorized: Client connection negotiation failed: client connection to 127.1.170.129:33649: server requires authentication, but client does not have Kerberos credentials available
I20260923 20:09:08.989982 1706 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqUuku6/build/asan/bin/kudu with pid 4229
I20260923 20:09:09.010442 1706 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskqUuku6/build/asan/bin/kudu pid 4363
W20260923 20:09:09.014564 1706 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskqUuku6/build/asan/bin/kudu pid 4363: Not authorized: Client connection negotiation failed: client connection to 127.1.170.130:44111: server requires authentication, but client does not have Kerberos credentials available
W20260923 20:09:09.014775 4638 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 20:09:09.011807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33668 (local address 127.1.170.130:44111)
0923 20:09:09.012815 (+ 1008us) server_negotiation.cc:206] Beginning negotiation
0923 20:09:09.012823 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 20:09:09.012854 (+ 31us) server_negotiation.cc:407] Connection header received
0923 20:09:09.012931 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:09:09.012947 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:09:09.013212 (+ 265us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:09:09.013387 (+ 175us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:09:09.014448 (+ 1061us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33668: BlockingRecv error: recv got EOF from 127.0.0.1:33668 (error 108)
Metrics: {"server-negotiator.queue_time_us":816,"thread_start_us":651,"threads_started":1}
I20260923 20:09:09.014959 1706 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqUuku6/build/asan/bin/kudu with pid 4363
I20260923 20:09:09.034154 1706 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskqUuku6/build/asan/bin/kudu pid 4497
W20260923 20:09:09.038398 1706 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskqUuku6/build/asan/bin/kudu pid 4497: Not authorized: Client connection negotiation failed: client connection to 127.1.170.131:35531: server requires authentication, but client does not have Kerberos credentials available
I20260923 20:09:09.038599 1706 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqUuku6/build/asan/bin/kudu with pid 4497
I20260923 20:09:09.056561 1706 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskqUuku6/build/asan/bin/kudu pid 4136
I20260923 20:09:09.057765 4198 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:09:09.172416 1706 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqUuku6/build/asan/bin/kudu with pid 4136
2026-09-23T20:09:09Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (5160 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790194121577027-1706-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 20:09:09 dist-test-slave-th4g krb5kdc[4643](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 20:09:09 dist-test-slave-th4g krb5kdc[4643](info): set up 2 sockets
Sep 23 20:09:09 dist-test-slave-th4g krb5kdc[4643](info): commencing operation
krb5kdc: starting...
W20260923 20:09:10.330507 1706 mini_kdc.cc:121] Time spent starting KDC: real 1.112s user 0.002s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790194121577027-1706-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790194121577027-1706-0/krb5kdc/test-user.keytab.
Sep 23 20:09:10 dist-test-slave-th4g krb5kdc[4643](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194150, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T20:09:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T20:09:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.170.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.170.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.170.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790194121577027-1706-0/krb5kdc/kudu_127.1.170.190.keytab.
Entry for principal kudu/127.1.170.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790194121577027-1706-0/krb5kdc/kudu_127.1.170.190.keytab.
I20260923 20:09:10.538987 1706 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqUuku6/build/asan/bin/kudu
/tmp/dist-test-taskqUuku6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790194121577027-1706-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790194121577027-1706-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790194121577027-1706-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790194121577027-1706-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.170.190:42713
--webserver_interface=127.1.170.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790194121577027-1706-0/krb5kdc/kudu_127.1.170.190.keytab
--principal=kudu/127.1.170.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.170.148:42945
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.170.190:42713
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790194121577027-1706-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790194121577027-1706-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790194121577027-1706-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:09:10.883476 4656 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:09:10.883780 4656 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:09:10.883951 4656 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:09:10.891367 4656 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 20:09:10.891487 4656 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:09:10.891546 4656 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:09:10.891669 4656 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 20:09:10.891763 4656 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 20:09:10.902009 4656 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.170.148:42945
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790194121577027-1706-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790194121577027-1706-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.170.190:42713
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790194121577027-1706-0/krb5kdc/kudu_127.1.170.190.keytab
--principal=kudu/127.1.170.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.170.190:42713
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790194121577027-1706-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.170.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790194121577027-1706-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:09:10.903357 4656 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:09:10.904629 4656 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:09:10.917119 4661 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:09:10.917274 4662 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:09:10.917296 4664 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:09:10.918560 4656 server_base.cc:1034] running on GCE node
I20260923 20:09:10.919406 4656 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:09:10.920936 4656 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:09:10.922186 4656 hybrid_clock.cc:648] HybridClock initialized: now 1790194150922188 us; error 86 us; skew 500 ppm
I20260923 20:09:10.922590 4656 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:09:10 dist-test-slave-th4g krb5kdc[4643](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194150, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.170.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:09:10.926067 4656 init.cc:377] Logged in from keytab as kudu/127.1.170.190@KRBTEST.COM (short username kudu)
I20260923 20:09:10.927759 4656 webserver.cc:460] Webserver started at http://127.1.170.190:43279/ using document root <none> and password file <none>
I20260923 20:09:10.928444 4656 fs_manager.cc:359] Metadata directory not provided
I20260923 20:09:10.928588 4656 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:09:10.928903 4656 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:09:10.931942 4656 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790194121577027-1706-0/minicluster-data/master-0/data/instance:
uuid: "62110f1140ea4e2ba15ba18f0a141e4b"
format_stamp: "Formatted at 2026-09-23 20:09:10 on dist-test-slave-th4g"
server_key: "5ed3808c6ccc2fc113c934bf4452d29b"
server_key_iv: "1ef3f99ed63c8ed639f21250463e168f"
server_key_version: "encryptionkey@0"
I20260923 20:09:10.932778 4656 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790194121577027-1706-0/minicluster-data/master-0/wal/instance:
uuid: "62110f1140ea4e2ba15ba18f0a141e4b"
format_stamp: "Formatted at 2026-09-23 20:09:10 on dist-test-slave-th4g"
server_key: "5ed3808c6ccc2fc113c934bf4452d29b"
server_key_iv: "1ef3f99ed63c8ed639f21250463e168f"
server_key_version: "encryptionkey@0"
I20260923 20:09:10.938549 4656 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260923 20:09:10.942584 4671 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:09:10.944028 4656 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260923 20:09:10.944229 4656 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790194121577027-1706-0/minicluster-data/master-0/data,/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790194121577027-1706-0/minicluster-data/master-0/wal
uuid: "62110f1140ea4e2ba15ba18f0a141e4b"
format_stamp: "Formatted at 2026-09-23 20:09:10 on dist-test-slave-th4g"
server_key: "5ed3808c6ccc2fc113c934bf4452d29b"
server_key_iv: "1ef3f99ed63c8ed639f21250463e168f"
server_key_version: "encryptionkey@0"
I20260923 20:09:10.944442 4656 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790194121577027-1706-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790194121577027-1706-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790194121577027-1706-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:09:10.972333 4656 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:09:10.977238 4656 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:09:10.977586 4656 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:09:10.996977 4656 rpc_server.cc:307] RPC server started. Bound to: 127.1.170.190:42713
I20260923 20:09:10.997013 4722 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.170.190:42713 every 8 connection(s)
I20260923 20:09:10.998499 4656 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790194121577027-1706-0/minicluster-data/master-0/data/info.pb
I20260923 20:09:10.999450 1706 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqUuku6/build/asan/bin/kudu as pid 4656
I20260923 20:09:10.999709 1706 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790194121577027-1706-0/minicluster-data/master-0/wal/instance
I20260923 20:09:11.000301 1706 external_mini_cluster.cc:1428] Setting key 74f9aaa646e605eb39e31e956e78f8b1
I20260923 20:09:11.003590 4723 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 23 20:09:11 dist-test-slave-th4g krb5kdc[4643](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194150, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.170.190@KRBTEST.COM
I20260923 20:09:11.014667 4723 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 62110f1140ea4e2ba15ba18f0a141e4b: Bootstrap starting.
I20260923 20:09:11.018551 4723 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 62110f1140ea4e2ba15ba18f0a141e4b: Neither blocks nor log segments found. Creating new log.
I20260923 20:09:11.019771 4723 log.cc:824] T 00000000000000000000000000000000 P 62110f1140ea4e2ba15ba18f0a141e4b: Log is configured to *not* fsync() on all Append() calls
I20260923 20:09:11.023241 4723 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 62110f1140ea4e2ba15ba18f0a141e4b: No bootstrap required, opened a new log
I20260923 20:09:11.023852 4726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:09:11.002259 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43788 (local address 127.1.170.190:42713)
0923 20:09:11.003071 (+ 812us) server_negotiation.cc:206] Beginning negotiation
0923 20:09:11.003083 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0923 20:09:11.003127 (+ 44us) server_negotiation.cc:407] Connection header received
0923 20:09:11.004877 (+ 1750us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:09:11.004899 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:09:11.005287 (+ 388us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:09:11.005682 (+ 395us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:09:11.007238 (+ 1556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:09:11.008664 (+ 1426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:09:11.009927 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:09:11.010588 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:09:11.010792 (+ 204us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:09:11.015500 (+ 4708us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:09:11.015508 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:09:11.015523 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:09:11.015568 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:09:11.019207 (+ 3639us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:09:11.020145 (+ 938us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:09:11.020152 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:09:11.020159 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:09:11.020480 (+ 321us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:09:11.020931 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:09:11.020937 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:09:11.020942 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:09:11.021579 (+ 637us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:09:11.021803 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0923 20:09:11.022403 (+ 600us) server_negotiation.cc:299] Negotiation successful
0923 20:09:11.022812 (+ 409us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":579,"thread_start_us":342,"threads_started":1}
I20260923 20:09:11.027961 1706 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 20:09:11.033660 4723 raft_consensus.cc:348] T 00000000000000000000000000000000 P 62110f1140ea4e2ba15ba18f0a141e4b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62110f1140ea4e2ba15ba18f0a141e4b" member_type: VOTER last_known_addr { host: "127.1.170.190" port: 42713 } }
I20260923 20:09:11.034238 4723 raft_consensus.cc:374] T 00000000000000000000000000000000 P 62110f1140ea4e2ba15ba18f0a141e4b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:09:11.034353 4723 raft_consensus.cc:729] T 00000000000000000000000000000000 P 62110f1140ea4e2ba15ba18f0a141e4b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 62110f1140ea4e2ba15ba18f0a141e4b, State: Initialized, Role: FOLLOWER
I20260923 20:09:11.035070 4723 consensus_queue.cc:260] T 00000000000000000000000000000000 P 62110f1140ea4e2ba15ba18f0a141e4b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62110f1140ea4e2ba15ba18f0a141e4b" member_type: VOTER last_known_addr { host: "127.1.170.190" port: 42713 } }
I20260923 20:09:11.035454 4723 raft_consensus.cc:388] T 00000000000000000000000000000000 P 62110f1140ea4e2ba15ba18f0a141e4b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 20:09:11.035606 4723 raft_consensus.cc:482] T 00000000000000000000000000000000 P 62110f1140ea4e2ba15ba18f0a141e4b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 20:09:11.035805 4723 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 62110f1140ea4e2ba15ba18f0a141e4b [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:09:11.038069 4723 raft_consensus.cc:504] T 00000000000000000000000000000000 P 62110f1140ea4e2ba15ba18f0a141e4b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62110f1140ea4e2ba15ba18f0a141e4b" member_type: VOTER last_known_addr { host: "127.1.170.190" port: 42713 } }
I20260923 20:09:11.038612 4723 leader_election.cc:302] T 00000000000000000000000000000000 P 62110f1140ea4e2ba15ba18f0a141e4b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 62110f1140ea4e2ba15ba18f0a141e4b; no voters:
I20260923 20:09:11.039286 4723 leader_election.cc:288] T 00000000000000000000000000000000 P 62110f1140ea4e2ba15ba18f0a141e4b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 20:09:11.039439 4729 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 62110f1140ea4e2ba15ba18f0a141e4b [term 1 FOLLOWER]: Leader election won for term 1
I20260923 20:09:11.040731 4729 raft_consensus.cc:686] T 00000000000000000000000000000000 P 62110f1140ea4e2ba15ba18f0a141e4b [term 1 LEADER]: Becoming Leader. State: Replica: 62110f1140ea4e2ba15ba18f0a141e4b, State: Running, Role: LEADER
I20260923 20:09:11.041412 4729 consensus_queue.cc:237] T 00000000000000000000000000000000 P 62110f1140ea4e2ba15ba18f0a141e4b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62110f1140ea4e2ba15ba18f0a141e4b" member_type: VOTER last_known_addr { host: "127.1.170.190" port: 42713 } }
I20260923 20:09:11.041591 4723 sys_catalog.cc:564] T 00000000000000000000000000000000 P 62110f1140ea4e2ba15ba18f0a141e4b [sys.catalog]: configured and running, proceeding with master startup.
I20260923 20:09:11.048795 4730 sys_catalog.cc:455] T 00000000000000000000000000000000 P 62110f1140ea4e2ba15ba18f0a141e4b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "62110f1140ea4e2ba15ba18f0a141e4b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62110f1140ea4e2ba15ba18f0a141e4b" member_type: VOTER last_known_addr { host: "127.1.170.190" port: 42713 } } }
I20260923 20:09:11.049192 4730 sys_catalog.cc:458] T 00000000000000000000000000000000 P 62110f1140ea4e2ba15ba18f0a141e4b [sys.catalog]: This master's current role is: LEADER
I20260923 20:09:11.048944 4731 sys_catalog.cc:455] T 00000000000000000000000000000000 P 62110f1140ea4e2ba15ba18f0a141e4b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 62110f1140ea4e2ba15ba18f0a141e4b. Latest consensus state: current_term: 1 leader_uuid: "62110f1140ea4e2ba15ba18f0a141e4b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62110f1140ea4e2ba15ba18f0a141e4b" member_type: VOTER last_known_addr { host: "127.1.170.190" port: 42713 } } }
I20260923 20:09:11.049329 4731 sys_catalog.cc:458] T 00000000000000000000000000000000 P 62110f1140ea4e2ba15ba18f0a141e4b [sys.catalog]: This master's current role is: LEADER
W20260923 20:09:11.060113 4744 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 62110f1140ea4e2ba15ba18f0a141e4b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260923 20:09:11.060289 4744 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260923 20:09:11.060408 4743 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 20:09:11.062865 4743 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 20:09:11.073395 4743 catalog_manager.cc:1347] Generated new cluster ID: 3bf575e72b194e5a9cfa088fa813e204
I20260923 20:09:11.073556 4743 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 20:09:11.082952 4726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:09:11.072518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43800 (local address 127.1.170.190:42713)
0923 20:09:11.072713 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0923 20:09:11.072720 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:09:11.074285 (+ 1565us) server_negotiation.cc:407] Connection header received
0923 20:09:11.074506 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:09:11.074517 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:09:11.074616 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:09:11.074745 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:09:11.075971 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:09:11.077177 (+ 1206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:09:11.078998 (+ 1821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:09:11.079668 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:09:11.079833 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:09:11.081009 (+ 1176us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:09:11.081016 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:09:11.081021 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 20:09:11.082032 (+ 1011us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:09:11.082089 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 20:09:11.082123 (+ 34us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:09:11.082251 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0923 20:09:11.082398 (+ 147us) server_negotiation.cc:299] Negotiation successful
0923 20:09:11.082504 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260923 20:09:11.086344 4743 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 20:09:11.087929 4743 catalog_manager.cc:1504] Loading token signing keys...
I20260923 20:09:11.099524 4743 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 62110f1140ea4e2ba15ba18f0a141e4b: Generated new TSK 0
I20260923 20:09:11.100687 4743 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 20:09:11.113488 1706 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskqUuku6/build/asan/bin/kudu pid 4656
I20260923 20:09:11.114832 4718 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:09:11.206035 1706 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqUuku6/build/asan/bin/kudu with pid 4656
2026-09-23T20:09:11Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (2026 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790194121577027-1706-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 20:09:11 dist-test-slave-th4g krb5kdc[4757](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 20:09:11 dist-test-slave-th4g krb5kdc[4757](info): set up 2 sockets
Sep 23 20:09:11 dist-test-slave-th4g krb5kdc[4757](info): commencing operation
krb5kdc: starting...
W20260923 20:09:12.331027 1706 mini_kdc.cc:121] Time spent starting KDC: real 1.088s user 0.000s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790194121577027-1706-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790194121577027-1706-0/krb5kdc/test-user.keytab.
Sep 23 20:09:12 dist-test-slave-th4g krb5kdc[4757](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194152, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T20:09:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T20:09:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.170.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.170.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.170.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790194121577027-1706-0/krb5kdc/kudu_127.1.170.190.keytab.
Entry for principal kudu/127.1.170.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790194121577027-1706-0/krb5kdc/kudu_127.1.170.190.keytab.
I20260923 20:09:12.551720 1706 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqUuku6/build/asan/bin/kudu
/tmp/dist-test-taskqUuku6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790194121577027-1706-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790194121577027-1706-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790194121577027-1706-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790194121577027-1706-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.170.190:46827
--webserver_interface=127.1.170.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790194121577027-1706-0/krb5kdc/kudu_127.1.170.190.keytab
--principal=kudu/127.1.170.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.170.148:44225
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.170.190:46827
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790194121577027-1706-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790194121577027-1706-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790194121577027-1706-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:09:12.895517 4770 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:09:12.895843 4770 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:09:12.895931 4770 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:09:12.903430 4770 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 20:09:12.903555 4770 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:09:12.903602 4770 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:09:12.903653 4770 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 20:09:12.903692 4770 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 20:09:12.915064 4770 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.170.148:44225
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790194121577027-1706-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790194121577027-1706-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.170.190:46827
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790194121577027-1706-0/krb5kdc/kudu_127.1.170.190.keytab
--principal=kudu/127.1.170.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.170.190:46827
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790194121577027-1706-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.170.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790194121577027-1706-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:09:12.916860 4770 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:09:12.918865 4770 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:09:12.934566 4778 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:09:12.934530 4776 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:09:12.934578 4775 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:09:12.935344 4770 server_base.cc:1034] running on GCE node
I20260923 20:09:12.936302 4770 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:09:12.938035 4770 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:09:12.939343 4770 hybrid_clock.cc:648] HybridClock initialized: now 1790194152939266 us; error 56 us; skew 500 ppm
I20260923 20:09:12.939770 4770 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:09:12 dist-test-slave-th4g krb5kdc[4757](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194152, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.170.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:09:12.943840 4770 init.cc:377] Logged in from keytab as kudu/127.1.170.190@KRBTEST.COM (short username kudu)
I20260923 20:09:12.945593 4770 webserver.cc:460] Webserver started at http://127.1.170.190:45769/ using document root <none> and password file <none>
I20260923 20:09:12.946511 4770 fs_manager.cc:359] Metadata directory not provided
I20260923 20:09:12.946682 4770 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:09:12.947054 4770 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:09:12.950397 4770 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790194121577027-1706-0/minicluster-data/master-0/data/instance:
uuid: "422d06d7b748434e83c14ec579b3ee73"
format_stamp: "Formatted at 2026-09-23 20:09:12 on dist-test-slave-th4g"
server_key: "e673bfca7bcca81c98e86624d1682d4c"
server_key_iv: "569e5bbb8aff26c3f884617119cce204"
server_key_version: "encryptionkey@0"
I20260923 20:09:12.951229 4770 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790194121577027-1706-0/minicluster-data/master-0/wal/instance:
uuid: "422d06d7b748434e83c14ec579b3ee73"
format_stamp: "Formatted at 2026-09-23 20:09:12 on dist-test-slave-th4g"
server_key: "e673bfca7bcca81c98e86624d1682d4c"
server_key_iv: "569e5bbb8aff26c3f884617119cce204"
server_key_version: "encryptionkey@0"
I20260923 20:09:12.956946 4770 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260923 20:09:12.960748 4785 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:09:12.962970 4770 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.002s
I20260923 20:09:12.963212 4770 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790194121577027-1706-0/minicluster-data/master-0/data,/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790194121577027-1706-0/minicluster-data/master-0/wal
uuid: "422d06d7b748434e83c14ec579b3ee73"
format_stamp: "Formatted at 2026-09-23 20:09:12 on dist-test-slave-th4g"
server_key: "e673bfca7bcca81c98e86624d1682d4c"
server_key_iv: "569e5bbb8aff26c3f884617119cce204"
server_key_version: "encryptionkey@0"
I20260923 20:09:12.963454 4770 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790194121577027-1706-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790194121577027-1706-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790194121577027-1706-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:09:13.015540 4770 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:09:13.020978 4770 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:09:13.021353 4770 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:09:13.043627 4770 rpc_server.cc:307] RPC server started. Bound to: 127.1.170.190:46827
I20260923 20:09:13.043627 4836 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.170.190:46827 every 8 connection(s)
I20260923 20:09:13.045130 4770 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790194121577027-1706-0/minicluster-data/master-0/data/info.pb
I20260923 20:09:13.049064 4837 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:09:13.054368 1706 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqUuku6/build/asan/bin/kudu as pid 4770
I20260923 20:09:13.054638 1706 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790194121577027-1706-0/minicluster-data/master-0/wal/instance
I20260923 20:09:13.055276 1706 external_mini_cluster.cc:1428] Setting key cc5995e051e68236b2c24c0efb420766
I20260923 20:09:13.061798 4837 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 422d06d7b748434e83c14ec579b3ee73: Bootstrap starting.
I20260923 20:09:13.065802 4837 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 422d06d7b748434e83c14ec579b3ee73: Neither blocks nor log segments found. Creating new log.
I20260923 20:09:13.067219 4837 log.cc:824] T 00000000000000000000000000000000 P 422d06d7b748434e83c14ec579b3ee73: Log is configured to *not* fsync() on all Append() calls
I20260923 20:09:13.071694 4837 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 422d06d7b748434e83c14ec579b3ee73: No bootstrap required, opened a new log
Sep 23 20:09:13 dist-test-slave-th4g krb5kdc[4757](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194152, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.170.190@KRBTEST.COM
I20260923 20:09:13.082867 4837 raft_consensus.cc:348] T 00000000000000000000000000000000 P 422d06d7b748434e83c14ec579b3ee73 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "422d06d7b748434e83c14ec579b3ee73" member_type: VOTER last_known_addr { host: "127.1.170.190" port: 46827 } }
I20260923 20:09:13.083387 4837 raft_consensus.cc:374] T 00000000000000000000000000000000 P 422d06d7b748434e83c14ec579b3ee73 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:09:13.083487 4837 raft_consensus.cc:729] T 00000000000000000000000000000000 P 422d06d7b748434e83c14ec579b3ee73 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 422d06d7b748434e83c14ec579b3ee73, State: Initialized, Role: FOLLOWER
I20260923 20:09:13.084209 4837 consensus_queue.cc:260] T 00000000000000000000000000000000 P 422d06d7b748434e83c14ec579b3ee73 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "422d06d7b748434e83c14ec579b3ee73" member_type: VOTER last_known_addr { host: "127.1.170.190" port: 46827 } }
I20260923 20:09:13.084486 4837 raft_consensus.cc:388] T 00000000000000000000000000000000 P 422d06d7b748434e83c14ec579b3ee73 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 20:09:13.084640 4837 raft_consensus.cc:482] T 00000000000000000000000000000000 P 422d06d7b748434e83c14ec579b3ee73 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 20:09:13.084847 4837 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 422d06d7b748434e83c14ec579b3ee73 [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:09:13.086751 4840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:09:13.058137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55060 (local address 127.1.170.190:46827)
0923 20:09:13.059371 (+ 1234us) server_negotiation.cc:206] Beginning negotiation
0923 20:09:13.059388 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0923 20:09:13.059443 (+ 55us) server_negotiation.cc:407] Connection header received
0923 20:09:13.061955 (+ 2512us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:09:13.061988 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:09:13.062555 (+ 567us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:09:13.063030 (+ 475us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:09:13.064453 (+ 1423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:09:13.066493 (+ 2040us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:09:13.068368 (+ 1875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:09:13.070053 (+ 1685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:09:13.070268 (+ 215us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:09:13.075591 (+ 5323us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:09:13.075613 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:09:13.075638 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:09:13.075716 (+ 78us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:09:13.080589 (+ 4873us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:09:13.081787 (+ 1198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:09:13.081798 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:09:13.081808 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:09:13.082077 (+ 269us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:09:13.082881 (+ 804us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:09:13.082889 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:09:13.082895 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:09:13.083660 (+ 765us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:09:13.083978 (+ 318us) server_negotiation.cc:1035] Waiting for connection context
0923 20:09:13.084711 (+ 733us) server_negotiation.cc:299] Negotiation successful
0923 20:09:13.085208 (+ 497us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":957,"thread_start_us":649,"threads_started":1}
I20260923 20:09:13.088701 4837 raft_consensus.cc:504] T 00000000000000000000000000000000 P 422d06d7b748434e83c14ec579b3ee73 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "422d06d7b748434e83c14ec579b3ee73" member_type: VOTER last_known_addr { host: "127.1.170.190" port: 46827 } }
I20260923 20:09:13.089363 4837 leader_election.cc:302] T 00000000000000000000000000000000 P 422d06d7b748434e83c14ec579b3ee73 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 422d06d7b748434e83c14ec579b3ee73; no voters:
I20260923 20:09:13.090337 4837 leader_election.cc:288] T 00000000000000000000000000000000 P 422d06d7b748434e83c14ec579b3ee73 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 20:09:13.090749 4842 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 422d06d7b748434e83c14ec579b3ee73 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 20:09:13.092736 4842 raft_consensus.cc:686] T 00000000000000000000000000000000 P 422d06d7b748434e83c14ec579b3ee73 [term 1 LEADER]: Becoming Leader. State: Replica: 422d06d7b748434e83c14ec579b3ee73, State: Running, Role: LEADER
I20260923 20:09:13.093308 4837 sys_catalog.cc:564] T 00000000000000000000000000000000 P 422d06d7b748434e83c14ec579b3ee73 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 20:09:13.093608 4842 consensus_queue.cc:237] T 00000000000000000000000000000000 P 422d06d7b748434e83c14ec579b3ee73 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "422d06d7b748434e83c14ec579b3ee73" member_type: VOTER last_known_addr { host: "127.1.170.190" port: 46827 } }
I20260923 20:09:13.098240 4843 sys_catalog.cc:455] T 00000000000000000000000000000000 P 422d06d7b748434e83c14ec579b3ee73 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "422d06d7b748434e83c14ec579b3ee73" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "422d06d7b748434e83c14ec579b3ee73" member_type: VOTER last_known_addr { host: "127.1.170.190" port: 46827 } } }
I20260923 20:09:13.098526 4844 sys_catalog.cc:455] T 00000000000000000000000000000000 P 422d06d7b748434e83c14ec579b3ee73 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 422d06d7b748434e83c14ec579b3ee73. Latest consensus state: current_term: 1 leader_uuid: "422d06d7b748434e83c14ec579b3ee73" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "422d06d7b748434e83c14ec579b3ee73" member_type: VOTER last_known_addr { host: "127.1.170.190" port: 46827 } } }
I20260923 20:09:13.098990 4843 sys_catalog.cc:458] T 00000000000000000000000000000000 P 422d06d7b748434e83c14ec579b3ee73 [sys.catalog]: This master's current role is: LEADER
I20260923 20:09:13.098999 4844 sys_catalog.cc:458] T 00000000000000000000000000000000 P 422d06d7b748434e83c14ec579b3ee73 [sys.catalog]: This master's current role is: LEADER
I20260923 20:09:13.102389 4853 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 20:09:13.107969 4853 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 20:09:13.117805 4853 catalog_manager.cc:1347] Generated new cluster ID: d340c48816544c05969cfe201c530f5f
I20260923 20:09:13.117964 4853 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 20:09:13.137456 4853 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 20:09:13.139758 4853 catalog_manager.cc:1504] Loading token signing keys...
I20260923 20:09:13.147835 4853 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 422d06d7b748434e83c14ec579b3ee73: Generated new TSK 0
I20260923 20:09:13.148753 4853 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.170.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.170.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.170.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790194121577027-1706-0/krb5kdc/kudu_127.1.170.129.keytab.
Entry for principal kudu/127.1.170.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790194121577027-1706-0/krb5kdc/kudu_127.1.170.129.keytab.
I20260923 20:09:13.211097 1706 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqUuku6/build/asan/bin/kudu
/tmp/dist-test-taskqUuku6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790194121577027-1706-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790194121577027-1706-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790194121577027-1706-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790194121577027-1706-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.170.129:0
--local_ip_for_outbound_sockets=127.1.170.129
--webserver_interface=127.1.170.129
--webserver_port=0
--tserver_master_addrs=127.1.170.190:46827
--keytab_file=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790194121577027-1706-0/krb5kdc/kudu_127.1.170.129.keytab
--principal=kudu/127.1.170.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.170.148:44225
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790194121577027-1706-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790194121577027-1706-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790194121577027-1706-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:09:13.561903 4863 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:09:13.562196 4863 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:09:13.562340 4863 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:09:13.569850 4863 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:09:13.569981 4863 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:09:13.570158 4863 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.170.129
I20260923 20:09:13.580106 4863 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.170.148:44225
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790194121577027-1706-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790194121577027-1706-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790194121577027-1706-0/krb5kdc/kudu_127.1.170.129.keytab
--principal=kudu/127.1.170.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.170.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790194121577027-1706-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.170.129
--webserver_port=0
--tserver_master_addrs=127.1.170.190:46827
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.170.129
--log_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790194121577027-1706-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:09:13.581499 4863 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:09:13.582858 4863 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:09:13.596626 4871 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:09:13.596783 4869 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:09:13.597057 4868 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:09:13.598268 4863 server_base.cc:1034] running on GCE node
I20260923 20:09:13.598946 4863 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:09:13.599975 4863 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:09:13.601235 4863 hybrid_clock.cc:648] HybridClock initialized: now 1790194153601149 us; error 48 us; skew 500 ppm
I20260923 20:09:13.601656 4863 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:09:13 dist-test-slave-th4g krb5kdc[4757](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194153, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.170.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:09:13.605059 4863 init.cc:377] Logged in from keytab as kudu/127.1.170.129@KRBTEST.COM (short username kudu)
I20260923 20:09:13.606643 4863 webserver.cc:460] Webserver started at http://127.1.170.129:41473/ using document root <none> and password file <none>
I20260923 20:09:13.607372 4863 fs_manager.cc:359] Metadata directory not provided
I20260923 20:09:13.607479 4863 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:09:13.607803 4863 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:09:13.610841 4863 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790194121577027-1706-0/minicluster-data/ts-0/data/instance:
uuid: "f1c54ccadf104d128a5a185b750e6fe1"
format_stamp: "Formatted at 2026-09-23 20:09:13 on dist-test-slave-th4g"
server_key: "bf790b005f6614c0631ad0b6f6f9c0fc"
server_key_iv: "272f3f94efa026c68922faf0514918a8"
server_key_version: "encryptionkey@0"
I20260923 20:09:13.611639 4863 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790194121577027-1706-0/minicluster-data/ts-0/wal/instance:
uuid: "f1c54ccadf104d128a5a185b750e6fe1"
format_stamp: "Formatted at 2026-09-23 20:09:13 on dist-test-slave-th4g"
server_key: "bf790b005f6614c0631ad0b6f6f9c0fc"
server_key_iv: "272f3f94efa026c68922faf0514918a8"
server_key_version: "encryptionkey@0"
I20260923 20:09:13.616803 4863 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260923 20:09:13.620431 4878 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:09:13.621732 4863 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260923 20:09:13.621927 4863 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790194121577027-1706-0/minicluster-data/ts-0/data,/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790194121577027-1706-0/minicluster-data/ts-0/wal
uuid: "f1c54ccadf104d128a5a185b750e6fe1"
format_stamp: "Formatted at 2026-09-23 20:09:13 on dist-test-slave-th4g"
server_key: "bf790b005f6614c0631ad0b6f6f9c0fc"
server_key_iv: "272f3f94efa026c68922faf0514918a8"
server_key_version: "encryptionkey@0"
I20260923 20:09:13.622130 4863 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790194121577027-1706-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790194121577027-1706-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790194121577027-1706-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:09:13.636054 4863 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:09:13.640314 4863 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:09:13.640729 4863 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:09:13.641709 4863 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 20:09:13.643240 4863 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 20:09:13.643318 4863 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 20:09:13.643432 4863 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 20:09:13.643496 4863 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 20:09:13.677575 4863 rpc_server.cc:307] RPC server started. Bound to: 127.1.170.129:34777
I20260923 20:09:13.677618 4990 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.170.129:34777 every 8 connection(s)
I20260923 20:09:13.678884 4863 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790194121577027-1706-0/minicluster-data/ts-0/data/info.pb
I20260923 20:09:13.684003 1706 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqUuku6/build/asan/bin/kudu as pid 4863
I20260923 20:09:13.684221 1706 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790194121577027-1706-0/minicluster-data/ts-0/wal/instance
I20260923 20:09:13.684896 1706 external_mini_cluster.cc:1428] Setting key 9553212a754c3eea4930fa9cdcd3ead6
Sep 23 20:09:13 dist-test-slave-th4g krb5kdc[4757](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194153, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.170.129@KRBTEST.COM for kudu/127.1.170.190@KRBTEST.COM
I20260923 20:09:13.701122 4993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:09:13.681650 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.170.129:44209 (local address 127.1.170.190:46827)
0923 20:09:13.682117 (+ 467us) server_negotiation.cc:206] Beginning negotiation
0923 20:09:13.682122 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 20:09:13.682993 (+ 871us) server_negotiation.cc:407] Connection header received
0923 20:09:13.684952 (+ 1959us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:09:13.684958 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:09:13.685051 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:09:13.685167 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:09:13.686872 (+ 1705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:09:13.687900 (+ 1028us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:09:13.689459 (+ 1559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:09:13.690197 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:09:13.690340 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:09:13.694162 (+ 3822us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:09:13.694171 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:09:13.694178 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:09:13.694229 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:09:13.697687 (+ 3458us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:09:13.698469 (+ 782us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:09:13.698477 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:09:13.698484 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:09:13.698622 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:09:13.699052 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:09:13.699060 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:09:13.699066 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:09:13.699465 (+ 399us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:09:13.699611 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0923 20:09:13.700404 (+ 793us) server_negotiation.cc:299] Negotiation successful
0923 20:09:13.700643 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":363,"thread_start_us":272,"threads_started":1}
I20260923 20:09:13.701721 4994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:09:13.682175 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.190:46827 (local address 127.1.170.129:44209)
0923 20:09:13.682845 (+ 670us) negotiation.cc:107] Waiting for socket to connect
0923 20:09:13.682880 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0923 20:09:13.684698 (+ 1818us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:09:13.685402 (+ 704us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:09:13.685432 (+ 30us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:09:13.686069 (+ 637us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:09:13.686679 (+ 610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:09:13.686702 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:09:13.688074 (+ 1372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:09:13.688078 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:09:13.689317 (+ 1239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:09:13.689328 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:09:13.690410 (+ 1082us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:09:13.690415 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:09:13.690601 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:09:13.691281 (+ 680us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:09:13.691302 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:09:13.693953 (+ 2651us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:09:13.697832 (+ 3879us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:09:13.697839 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:09:13.697852 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:09:13.698338 (+ 486us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:09:13.698726 (+ 388us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:09:13.698730 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:09:13.698732 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:09:13.698928 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:09:13.699590 (+ 662us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:09:13.699596 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:09:13.699966 (+ 370us) client_negotiation.cc:769] Sending connection context
0923 20:09:13.700235 (+ 269us) client_negotiation.cc:240] Negotiation successful
0923 20:09:13.700626 (+ 391us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":460,"thread_start_us":251,"threads_started":1}
I20260923 20:09:13.703645 4991 heartbeater.cc:347] Connected to a master server at 127.1.170.190:46827
I20260923 20:09:13.703949 4991 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.1.170.130@KRBTEST.COM; defaulting to no policy
I20260923 20:09:13.704723 4991 heartbeater.cc:511] Master 127.1.170.190:46827 requested a full tablet report, sending...
I20260923 20:09:13.707006 4802 ts_manager.cc:194] Registered new tserver with Master: f1c54ccadf104d128a5a185b750e6fe1 (127.1.170.129:34777)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.170.130@KRBTEST.COM" created.
I20260923 20:09:13.709142 4802 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.170.129@KRBTEST.COM'} at 127.1.170.129:44209
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.170.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790194121577027-1706-0/krb5kdc/kudu_127.1.170.130.keytab.
Entry for principal kudu/127.1.170.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790194121577027-1706-0/krb5kdc/kudu_127.1.170.130.keytab.
I20260923 20:09:13.734095 1706 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqUuku6/build/asan/bin/kudu
/tmp/dist-test-taskqUuku6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790194121577027-1706-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790194121577027-1706-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790194121577027-1706-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790194121577027-1706-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.170.130:0
--local_ip_for_outbound_sockets=127.1.170.130
--webserver_interface=127.1.170.130
--webserver_port=0
--tserver_master_addrs=127.1.170.190:46827
--keytab_file=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790194121577027-1706-0/krb5kdc/kudu_127.1.170.130.keytab
--principal=kudu/127.1.170.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.170.148:44225
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790194121577027-1706-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790194121577027-1706-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790194121577027-1706-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:09:14.070112 4997 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:09:14.070447 4997 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:09:14.070614 4997 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:09:14.078136 4997 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:09:14.078316 4997 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:09:14.078490 4997 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.170.130
I20260923 20:09:14.088789 4997 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.170.148:44225
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790194121577027-1706-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790194121577027-1706-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790194121577027-1706-0/krb5kdc/kudu_127.1.170.130.keytab
--principal=kudu/127.1.170.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.170.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790194121577027-1706-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.170.130
--webserver_port=0
--tserver_master_addrs=127.1.170.190:46827
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.170.130
--log_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790194121577027-1706-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:09:14.090232 4997 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:09:14.091661 4997 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:09:14.104981 5005 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:09:14.104916 5003 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:09:14.105002 5002 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:09:14.105742 4997 server_base.cc:1034] running on GCE node
I20260923 20:09:14.106362 4997 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:09:14.107311 4997 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:09:14.108553 4997 hybrid_clock.cc:648] HybridClock initialized: now 1790194154108491 us; error 59 us; skew 500 ppm
I20260923 20:09:14.108879 4997 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:09:14 dist-test-slave-th4g krb5kdc[4757](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194154, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.170.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:09:14.118103 4997 init.cc:377] Logged in from keytab as kudu/127.1.170.130@KRBTEST.COM (short username kudu)
I20260923 20:09:14.119755 4997 webserver.cc:460] Webserver started at http://127.1.170.130:41445/ using document root <none> and password file <none>
I20260923 20:09:14.120494 4997 fs_manager.cc:359] Metadata directory not provided
I20260923 20:09:14.120647 4997 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:09:14.120957 4997 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:09:14.123951 4997 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790194121577027-1706-0/minicluster-data/ts-1/data/instance:
uuid: "d8d1b4f793e64df7bbb88aab8c92a601"
format_stamp: "Formatted at 2026-09-23 20:09:14 on dist-test-slave-th4g"
server_key: "1513bc9cd7536213ca8aea00810117c0"
server_key_iv: "c1ff0c623a1b0212dfd21fbdedcfe9d7"
server_key_version: "encryptionkey@0"
I20260923 20:09:14.124812 4997 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790194121577027-1706-0/minicluster-data/ts-1/wal/instance:
uuid: "d8d1b4f793e64df7bbb88aab8c92a601"
format_stamp: "Formatted at 2026-09-23 20:09:14 on dist-test-slave-th4g"
server_key: "1513bc9cd7536213ca8aea00810117c0"
server_key_iv: "c1ff0c623a1b0212dfd21fbdedcfe9d7"
server_key_version: "encryptionkey@0"
I20260923 20:09:14.130364 4997 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.004s
I20260923 20:09:14.134212 5012 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:09:14.135607 4997 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260923 20:09:14.135882 4997 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790194121577027-1706-0/minicluster-data/ts-1/data,/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790194121577027-1706-0/minicluster-data/ts-1/wal
uuid: "d8d1b4f793e64df7bbb88aab8c92a601"
format_stamp: "Formatted at 2026-09-23 20:09:14 on dist-test-slave-th4g"
server_key: "1513bc9cd7536213ca8aea00810117c0"
server_key_iv: "c1ff0c623a1b0212dfd21fbdedcfe9d7"
server_key_version: "encryptionkey@0"
I20260923 20:09:14.136145 4997 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790194121577027-1706-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790194121577027-1706-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790194121577027-1706-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:09:14.157008 4997 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:09:14.161331 4997 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:09:14.161655 4997 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:09:14.162732 4997 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 20:09:14.164080 4997 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 20:09:14.164191 4997 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 20:09:14.164351 4997 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 20:09:14.164424 4997 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 20:09:14.199925 4997 rpc_server.cc:307] RPC server started. Bound to: 127.1.170.130:46625
I20260923 20:09:14.199966 5124 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.170.130:46625 every 8 connection(s)
I20260923 20:09:14.201200 4997 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790194121577027-1706-0/minicluster-data/ts-1/data/info.pb
I20260923 20:09:14.203435 1706 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqUuku6/build/asan/bin/kudu as pid 4997
I20260923 20:09:14.203635 1706 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790194121577027-1706-0/minicluster-data/ts-1/wal/instance
I20260923 20:09:14.204196 1706 external_mini_cluster.cc:1428] Setting key 3f3996b6fd794839e0a0c02aab2b3dea
Sep 23 20:09:14 dist-test-slave-th4g krb5kdc[4757](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194154, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.170.130@KRBTEST.COM for kudu/127.1.170.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.1.170.131@KRBTEST.COM; defaulting to no policy
I20260923 20:09:14.222972 5127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:09:14.204066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.170.130:59461 (local address 127.1.170.190:46827)
0923 20:09:14.204553 (+ 487us) server_negotiation.cc:206] Beginning negotiation
0923 20:09:14.204557 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 20:09:14.205664 (+ 1107us) server_negotiation.cc:407] Connection header received
0923 20:09:14.207678 (+ 2014us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:09:14.207684 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:09:14.207770 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:09:14.207886 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:09:14.209305 (+ 1419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:09:14.210096 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:09:14.211946 (+ 1850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:09:14.212424 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:09:14.212534 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:09:14.216819 (+ 4285us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:09:14.216824 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:09:14.216827 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:09:14.216859 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:09:14.219693 (+ 2834us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:09:14.220542 (+ 849us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:09:14.220549 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:09:14.220556 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:09:14.220692 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:09:14.221124 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:09:14.221131 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:09:14.221138 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:09:14.221406 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:09:14.221560 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0923 20:09:14.222351 (+ 791us) server_negotiation.cc:299] Negotiation successful
0923 20:09:14.222560 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":373,"thread_start_us":281,"threads_started":1}
I20260923 20:09:14.224006 5128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:09:14.204659 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.190:46827 (local address 127.1.170.130:59461)
0923 20:09:14.205490 (+ 831us) negotiation.cc:107] Waiting for socket to connect
0923 20:09:14.205545 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0923 20:09:14.207431 (+ 1886us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:09:14.208112 (+ 681us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:09:14.208120 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:09:14.208597 (+ 477us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:09:14.209136 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:09:14.209151 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:09:14.210235 (+ 1084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:09:14.210243 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:09:14.211757 (+ 1514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:09:14.211777 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:09:14.212660 (+ 883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:09:14.212670 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:09:14.213001 (+ 331us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:09:14.213790 (+ 789us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:09:14.213828 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:09:14.216538 (+ 2710us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:09:14.219945 (+ 3407us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:09:14.219955 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:09:14.219973 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:09:14.220415 (+ 442us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:09:14.220815 (+ 400us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:09:14.220820 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:09:14.220824 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:09:14.221006 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:09:14.221538 (+ 532us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:09:14.221546 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:09:14.221948 (+ 402us) client_negotiation.cc:769] Sending connection context
0923 20:09:14.222268 (+ 320us) client_negotiation.cc:240] Negotiation successful
0923 20:09:14.222800 (+ 532us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":536,"thread_start_us":325,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.170.131@KRBTEST.COM" created.
I20260923 20:09:14.225724 5125 heartbeater.cc:347] Connected to a master server at 127.1.170.190:46827
I20260923 20:09:14.226111 5125 heartbeater.cc:464] Registering TS with master...
I20260923 20:09:14.226915 5125 heartbeater.cc:511] Master 127.1.170.190:46827 requested a full tablet report, sending...
I20260923 20:09:14.228463 4802 ts_manager.cc:194] Registered new tserver with Master: d8d1b4f793e64df7bbb88aab8c92a601 (127.1.170.130:46625)
I20260923 20:09:14.229586 4802 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.170.130@KRBTEST.COM'} at 127.1.170.130:59461
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.170.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790194121577027-1706-0/krb5kdc/kudu_127.1.170.131.keytab.
Entry for principal kudu/127.1.170.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790194121577027-1706-0/krb5kdc/kudu_127.1.170.131.keytab.
I20260923 20:09:14.251116 1706 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqUuku6/build/asan/bin/kudu
/tmp/dist-test-taskqUuku6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790194121577027-1706-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790194121577027-1706-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790194121577027-1706-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790194121577027-1706-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.170.131:0
--local_ip_for_outbound_sockets=127.1.170.131
--webserver_interface=127.1.170.131
--webserver_port=0
--tserver_master_addrs=127.1.170.190:46827
--keytab_file=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790194121577027-1706-0/krb5kdc/kudu_127.1.170.131.keytab
--principal=kudu/127.1.170.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.170.148:44225
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790194121577027-1706-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790194121577027-1706-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790194121577027-1706-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:09:14.572597 5131 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:09:14.572904 5131 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:09:14.573000 5131 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:09:14.580265 5131 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:09:14.580394 5131 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:09:14.580511 5131 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.170.131
I20260923 20:09:14.590488 5131 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.170.148:44225
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790194121577027-1706-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790194121577027-1706-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790194121577027-1706-0/krb5kdc/kudu_127.1.170.131.keytab
--principal=kudu/127.1.170.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.170.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790194121577027-1706-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.170.131
--webserver_port=0
--tserver_master_addrs=127.1.170.190:46827
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.170.131
--log_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790194121577027-1706-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:09:14.591823 5131 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:09:14.593158 5131 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:09:14.608101 5139 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:09:14.608325 5136 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:09:14.607982 5137 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:09:14.612035 5131 server_base.cc:1034] running on GCE node
I20260923 20:09:14.612772 5131 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:09:14.613794 5131 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:09:14.615061 5131 hybrid_clock.cc:648] HybridClock initialized: now 1790194154615016 us; error 36 us; skew 500 ppm
I20260923 20:09:14.615417 5131 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:09:14 dist-test-slave-th4g krb5kdc[4757](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194154, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.170.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:09:14.618743 5131 init.cc:377] Logged in from keytab as kudu/127.1.170.131@KRBTEST.COM (short username kudu)
I20260923 20:09:14.620190 5131 webserver.cc:460] Webserver started at http://127.1.170.131:44079/ using document root <none> and password file <none>
I20260923 20:09:14.620877 5131 fs_manager.cc:359] Metadata directory not provided
I20260923 20:09:14.620971 5131 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:09:14.621207 5131 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:09:14.624044 5131 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790194121577027-1706-0/minicluster-data/ts-2/data/instance:
uuid: "d862652a8f624b70a21e998635fa7c25"
format_stamp: "Formatted at 2026-09-23 20:09:14 on dist-test-slave-th4g"
server_key: "b9d892338161e80d17cca2a29f3dd2e7"
server_key_iv: "ae7178c53be6b57d3dc69cc1f4dc8509"
server_key_version: "encryptionkey@0"
I20260923 20:09:14.624789 5131 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790194121577027-1706-0/minicluster-data/ts-2/wal/instance:
uuid: "d862652a8f624b70a21e998635fa7c25"
format_stamp: "Formatted at 2026-09-23 20:09:14 on dist-test-slave-th4g"
server_key: "b9d892338161e80d17cca2a29f3dd2e7"
server_key_iv: "ae7178c53be6b57d3dc69cc1f4dc8509"
server_key_version: "encryptionkey@0"
I20260923 20:09:14.630259 5131 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260923 20:09:14.633709 5146 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:09:14.635157 5131 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 20:09:14.635294 5131 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790194121577027-1706-0/minicluster-data/ts-2/data,/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790194121577027-1706-0/minicluster-data/ts-2/wal
uuid: "d862652a8f624b70a21e998635fa7c25"
format_stamp: "Formatted at 2026-09-23 20:09:14 on dist-test-slave-th4g"
server_key: "b9d892338161e80d17cca2a29f3dd2e7"
server_key_iv: "ae7178c53be6b57d3dc69cc1f4dc8509"
server_key_version: "encryptionkey@0"
I20260923 20:09:14.635463 5131 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790194121577027-1706-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790194121577027-1706-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790194121577027-1706-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:09:14.663168 5131 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:09:14.667362 5131 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:09:14.667624 5131 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:09:14.668689 5131 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 20:09:14.670038 5131 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 20:09:14.670113 5131 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 20:09:14.670195 5131 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 20:09:14.670230 5131 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 20:09:14.704154 5131 rpc_server.cc:307] RPC server started. Bound to: 127.1.170.131:43801
I20260923 20:09:14.704200 5258 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.170.131:43801 every 8 connection(s)
I20260923 20:09:14.705400 5131 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790194121577027-1706-0/minicluster-data/ts-2/data/info.pb
I20260923 20:09:14.710143 1706 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqUuku6/build/asan/bin/kudu as pid 5131
I20260923 20:09:14.710350 1706 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790194121577027-1706-0/minicluster-data/ts-2/wal/instance
I20260923 20:09:14.710760 1706 external_mini_cluster.cc:1428] Setting key 93f2b819ab4bc2273de68888b517f8cd
I20260923 20:09:14.712965 4991 heartbeater.cc:503] Master 127.1.170.190:46827 was elected leader, sending a full tablet report...
Sep 23 20:09:14 dist-test-slave-th4g krb5kdc[4757](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194154, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.170.131@KRBTEST.COM for kudu/127.1.170.190@KRBTEST.COM
I20260923 20:09:14.726037 5127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:09:14.707988 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.170.131:55337 (local address 127.1.170.190:46827)
0923 20:09:14.708199 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0923 20:09:14.708203 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 20:09:14.709254 (+ 1051us) server_negotiation.cc:407] Connection header received
0923 20:09:14.711258 (+ 2004us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:09:14.711263 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:09:14.711411 (+ 148us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:09:14.711517 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:09:14.713455 (+ 1938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:09:14.714433 (+ 978us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:09:14.715954 (+ 1521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:09:14.716343 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:09:14.716471 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:09:14.719937 (+ 3466us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:09:14.719942 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:09:14.719946 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:09:14.719975 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:09:14.722646 (+ 2671us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:09:14.723385 (+ 739us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:09:14.723390 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:09:14.723394 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:09:14.723511 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:09:14.723926 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:09:14.723933 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:09:14.723938 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:09:14.724299 (+ 361us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:09:14.724456 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0923 20:09:14.725419 (+ 963us) server_negotiation.cc:299] Negotiation successful
0923 20:09:14.725670 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260923 20:09:14.727275 5261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:09:14.708371 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.190:46827 (local address 127.1.170.131:55337)
0923 20:09:14.709073 (+ 702us) negotiation.cc:107] Waiting for socket to connect
0923 20:09:14.709114 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0923 20:09:14.710983 (+ 1869us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:09:14.711782 (+ 799us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:09:14.711793 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:09:14.712475 (+ 682us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:09:14.713290 (+ 815us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:09:14.713317 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:09:14.714573 (+ 1256us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:09:14.714578 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:09:14.715796 (+ 1218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:09:14.715811 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:09:14.716553 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:09:14.716559 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:09:14.716788 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:09:14.717466 (+ 678us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:09:14.717488 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:09:14.719719 (+ 2231us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:09:14.722800 (+ 3081us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:09:14.722809 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:09:14.722826 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:09:14.723254 (+ 428us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:09:14.723624 (+ 370us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:09:14.723629 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:09:14.723633 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:09:14.723815 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:09:14.724463 (+ 648us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:09:14.724471 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:09:14.724929 (+ 458us) client_negotiation.cc:769] Sending connection context
0923 20:09:14.725347 (+ 418us) client_negotiation.cc:240] Negotiation successful
0923 20:09:14.725994 (+ 647us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":482,"thread_start_us":286,"threads_started":1}
I20260923 20:09:14.728561 5259 heartbeater.cc:347] Connected to a master server at 127.1.170.190:46827
I20260923 20:09:14.728835 5259 heartbeater.cc:464] Registering TS with master...
I20260923 20:09:14.729522 5259 heartbeater.cc:511] Master 127.1.170.190:46827 requested a full tablet report, sending...
I20260923 20:09:14.730870 4802 ts_manager.cc:194] Registered new tserver with Master: d862652a8f624b70a21e998635fa7c25 (127.1.170.131:43801)
I20260923 20:09:14.731667 4802 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.170.131@KRBTEST.COM'} at 127.1.170.131:55337
I20260923 20:09:14.739236 1706 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 23 20:09:14 dist-test-slave-th4g krb5kdc[4757](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194154, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Sep 23 20:09:14 dist-test-slave-th4g krb5kdc[4757](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194154, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.1.170.190@KRBTEST.COM
I20260923 20:09:14.787227 5127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:09:14.773115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56322 (local address 127.1.170.190:46827)
0923 20:09:14.773291 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0923 20:09:14.773296 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 20:09:14.773798 (+ 502us) server_negotiation.cc:407] Connection header received
0923 20:09:14.773997 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:09:14.774001 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:09:14.774078 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:09:14.774167 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:09:14.775495 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:09:14.776295 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:09:14.777518 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:09:14.778095 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:09:14.778200 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:09:14.782165 (+ 3965us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:09:14.782170 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:09:14.782173 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:09:14.782209 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:09:14.784633 (+ 2424us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:09:14.785422 (+ 789us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:09:14.785426 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:09:14.785429 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:09:14.785541 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:09:14.785940 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:09:14.785948 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:09:14.785951 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:09:14.786288 (+ 337us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:09:14.786476 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0923 20:09:14.786803 (+ 327us) server_negotiation.cc:299] Negotiation successful
0923 20:09:14.786961 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
W20260923 20:09:14.787774 4802 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:56322
I20260923 20:09:14.789167 1706 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskqUuku6/build/asan/bin/kudu pid 4863
Sep 23 20:09:14 dist-test-slave-th4g krb5kdc[4757](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194154, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.1.170.129@KRBTEST.COM
I20260923 20:09:14.805565 5270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:09:14.790602 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34016 (local address 127.1.170.129:34777)
0923 20:09:14.791193 (+ 591us) server_negotiation.cc:206] Beginning negotiation
0923 20:09:14.791203 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0923 20:09:14.791227 (+ 24us) server_negotiation.cc:407] Connection header received
0923 20:09:14.791308 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:09:14.791315 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:09:14.791511 (+ 196us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:09:14.791689 (+ 178us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:09:14.792859 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:09:14.794010 (+ 1151us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:09:14.795197 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:09:14.795763 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:09:14.795903 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:09:14.799779 (+ 3876us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:09:14.799789 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:09:14.799803 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:09:14.799843 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:09:14.802774 (+ 2931us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:09:14.803459 (+ 685us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:09:14.803472 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:09:14.803474 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:09:14.803594 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:09:14.803994 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:09:14.804000 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:09:14.804005 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:09:14.804347 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:09:14.804670 (+ 323us) server_negotiation.cc:1035] Waiting for connection context
0923 20:09:14.804938 (+ 268us) server_negotiation.cc:299] Negotiation successful
0923 20:09:14.805209 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":419,"thread_start_us":258,"threads_started":1}
W20260923 20:09:14.806929 4986 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:34016
W20260923 20:09:14.808068 1706 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskqUuku6/build/asan/bin/kudu pid 4863: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260923 20:09:14.808267 1706 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqUuku6/build/asan/bin/kudu with pid 4863
I20260923 20:09:14.824082 1706 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskqUuku6/build/asan/bin/kudu pid 4997
Sep 23 20:09:14 dist-test-slave-th4g krb5kdc[4757](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194154, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.1.170.130@KRBTEST.COM
I20260923 20:09:14.840296 5271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:09:14.825060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42946 (local address 127.1.170.130:46625)
0923 20:09:14.825572 (+ 512us) server_negotiation.cc:206] Beginning negotiation
0923 20:09:14.825586 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0923 20:09:14.825611 (+ 25us) server_negotiation.cc:407] Connection header received
0923 20:09:14.825670 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:09:14.825676 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:09:14.825952 (+ 276us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:09:14.826081 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:09:14.827354 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:09:14.828367 (+ 1013us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:09:14.829614 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:09:14.830294 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:09:14.830398 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:09:14.834499 (+ 4101us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:09:14.834510 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:09:14.834525 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:09:14.834558 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:09:14.837329 (+ 2771us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:09:14.838107 (+ 778us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:09:14.838114 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:09:14.838117 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:09:14.838261 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:09:14.838589 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:09:14.838592 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:09:14.838594 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:09:14.838969 (+ 375us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:09:14.839228 (+ 259us) server_negotiation.cc:1035] Waiting for connection context
0923 20:09:14.839692 (+ 464us) server_negotiation.cc:299] Negotiation successful
0923 20:09:14.839894 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":376,"thread_start_us":272,"threads_started":1}
W20260923 20:09:14.841675 5120 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:42946
W20260923 20:09:14.842760 1706 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskqUuku6/build/asan/bin/kudu pid 4997: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260923 20:09:14.842906 1706 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqUuku6/build/asan/bin/kudu with pid 4997
I20260923 20:09:14.858098 1706 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskqUuku6/build/asan/bin/kudu pid 5131
Sep 23 20:09:14 dist-test-slave-th4g krb5kdc[4757](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194154, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.1.170.131@KRBTEST.COM
I20260923 20:09:14.875989 5272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:09:14.858998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32768 (local address 127.1.170.131:43801)
0923 20:09:14.860034 (+ 1036us) server_negotiation.cc:206] Beginning negotiation
0923 20:09:14.860043 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 20:09:14.860086 (+ 43us) server_negotiation.cc:407] Connection header received
0923 20:09:14.860169 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:09:14.860177 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:09:14.860604 (+ 427us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:09:14.860827 (+ 223us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:09:14.862253 (+ 1426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:09:14.863327 (+ 1074us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:09:14.864519 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:09:14.865083 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:09:14.865233 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:09:14.869840 (+ 4607us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:09:14.869852 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:09:14.869868 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:09:14.870031 (+ 163us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:09:14.872886 (+ 2855us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:09:14.873628 (+ 742us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:09:14.873635 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:09:14.873637 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:09:14.873761 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:09:14.874155 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:09:14.874158 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:09:14.874160 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:09:14.874519 (+ 359us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:09:14.874753 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0923 20:09:14.875402 (+ 649us) server_negotiation.cc:299] Negotiation successful
0923 20:09:14.875619 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":825,"thread_start_us":605,"threads_started":1}
W20260923 20:09:14.877784 5254 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:32768
W20260923 20:09:14.879235 1706 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskqUuku6/build/asan/bin/kudu pid 5131: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260923 20:09:14.879400 1706 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqUuku6/build/asan/bin/kudu with pid 5131
I20260923 20:09:14.897243 1706 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskqUuku6/build/asan/bin/kudu pid 4770
I20260923 20:09:14.898518 4832 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:09:15.001543 1706 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqUuku6/build/asan/bin/kudu with pid 4770
2026-09-23T20:09:15Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (3800 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790194121577027-1706-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 20:09:15 dist-test-slave-th4g krb5kdc[5276](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 20:09:15 dist-test-slave-th4g krb5kdc[5276](info): set up 2 sockets
Sep 23 20:09:15 dist-test-slave-th4g krb5kdc[5276](info): commencing operation
krb5kdc: starting...
W20260923 20:09:16.134766 1706 mini_kdc.cc:121] Time spent starting KDC: real 1.091s user 0.000s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790194121577027-1706-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790194121577027-1706-0/krb5kdc/test-user.keytab.
Sep 23 20:09:16 dist-test-slave-th4g krb5kdc[5276](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194156, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T20:09:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T20:09:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.170.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.170.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.170.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790194121577027-1706-0/krb5kdc/kudu_127.1.170.190.keytab.
Entry for principal kudu/127.1.170.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790194121577027-1706-0/krb5kdc/kudu_127.1.170.190.keytab.
I20260923 20:09:16.368557 1706 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqUuku6/build/asan/bin/kudu
/tmp/dist-test-taskqUuku6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790194121577027-1706-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790194121577027-1706-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790194121577027-1706-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790194121577027-1706-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.170.190:32911
--webserver_interface=127.1.170.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790194121577027-1706-0/krb5kdc/kudu_127.1.170.190.keytab
--principal=kudu/127.1.170.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.170.148:43449
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.170.190:32911
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790194121577027-1706-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790194121577027-1706-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790194121577027-1706-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:09:16.723469 5289 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:09:16.723845 5289 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:09:16.724000 5289 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:09:16.731417 5289 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 20:09:16.731609 5289 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:09:16.731720 5289 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:09:16.731833 5289 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 20:09:16.731911 5289 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 20:09:16.744678 5289 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.170.148:43449
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790194121577027-1706-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790194121577027-1706-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.170.190:32911
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790194121577027-1706-0/krb5kdc/kudu_127.1.170.190.keytab
--principal=kudu/127.1.170.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.170.190:32911
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790194121577027-1706-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.170.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790194121577027-1706-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:09:16.746181 5289 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:09:16.747553 5289 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:09:16.762280 5295 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:09:16.762310 5297 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:09:16.762257 5294 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:09:16.763310 5289 server_base.cc:1034] running on GCE node
I20260923 20:09:16.764235 5289 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:09:16.765936 5289 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:09:16.767192 5289 hybrid_clock.cc:648] HybridClock initialized: now 1790194156767149 us; error 44 us; skew 500 ppm
I20260923 20:09:16.767592 5289 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:09:16 dist-test-slave-th4g krb5kdc[5276](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194156, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.170.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:09:16.772148 5289 init.cc:377] Logged in from keytab as kudu/127.1.170.190@KRBTEST.COM (short username kudu)
I20260923 20:09:16.774384 5289 webserver.cc:460] Webserver started at http://127.1.170.190:43187/ using document root <none> and password file <none>
I20260923 20:09:16.775170 5289 fs_manager.cc:359] Metadata directory not provided
I20260923 20:09:16.775323 5289 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:09:16.775720 5289 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:09:16.779042 5289 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790194121577027-1706-0/minicluster-data/master-0/data/instance:
uuid: "a02c794cff2d4fa89c47869d1107b5b5"
format_stamp: "Formatted at 2026-09-23 20:09:16 on dist-test-slave-th4g"
server_key: "c622a52f6c00553d547ebeccc0e751b2"
server_key_iv: "f81eaf3ec73e66d79e3e58e32755a797"
server_key_version: "encryptionkey@0"
I20260923 20:09:16.779909 5289 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790194121577027-1706-0/minicluster-data/master-0/wal/instance:
uuid: "a02c794cff2d4fa89c47869d1107b5b5"
format_stamp: "Formatted at 2026-09-23 20:09:16 on dist-test-slave-th4g"
server_key: "c622a52f6c00553d547ebeccc0e751b2"
server_key_iv: "f81eaf3ec73e66d79e3e58e32755a797"
server_key_version: "encryptionkey@0"
I20260923 20:09:16.786379 5289 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260923 20:09:16.790434 5304 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:09:16.792115 5289 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260923 20:09:16.792361 5289 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790194121577027-1706-0/minicluster-data/master-0/data,/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790194121577027-1706-0/minicluster-data/master-0/wal
uuid: "a02c794cff2d4fa89c47869d1107b5b5"
format_stamp: "Formatted at 2026-09-23 20:09:16 on dist-test-slave-th4g"
server_key: "c622a52f6c00553d547ebeccc0e751b2"
server_key_iv: "f81eaf3ec73e66d79e3e58e32755a797"
server_key_version: "encryptionkey@0"
I20260923 20:09:16.792601 5289 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790194121577027-1706-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790194121577027-1706-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790194121577027-1706-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:09:16.808670 5289 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:09:16.813747 5289 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:09:16.814132 5289 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:09:16.833022 5289 rpc_server.cc:307] RPC server started. Bound to: 127.1.170.190:32911
I20260923 20:09:16.833051 5355 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.170.190:32911 every 8 connection(s)
I20260923 20:09:16.834549 5289 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790194121577027-1706-0/minicluster-data/master-0/data/info.pb
I20260923 20:09:16.838459 5356 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:09:16.840799 1706 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqUuku6/build/asan/bin/kudu as pid 5289
I20260923 20:09:16.841005 1706 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790194121577027-1706-0/minicluster-data/master-0/wal/instance
I20260923 20:09:16.841641 1706 external_mini_cluster.cc:1428] Setting key ec088f05462a7f177e5494e6eacd7b98
I20260923 20:09:16.850715 5356 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a02c794cff2d4fa89c47869d1107b5b5: Bootstrap starting.
I20260923 20:09:16.854966 5356 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a02c794cff2d4fa89c47869d1107b5b5: Neither blocks nor log segments found. Creating new log.
I20260923 20:09:16.856487 5356 log.cc:824] T 00000000000000000000000000000000 P a02c794cff2d4fa89c47869d1107b5b5: Log is configured to *not* fsync() on all Append() calls
Sep 23 20:09:16 dist-test-slave-th4g krb5kdc[5276](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194156, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.170.190@KRBTEST.COM
I20260923 20:09:16.859889 5356 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a02c794cff2d4fa89c47869d1107b5b5: No bootstrap required, opened a new log
I20260923 20:09:16.866513 5359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:09:16.843706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38426 (local address 127.1.170.190:32911)
0923 20:09:16.844503 (+ 797us) server_negotiation.cc:206] Beginning negotiation
0923 20:09:16.844515 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0923 20:09:16.844556 (+ 41us) server_negotiation.cc:407] Connection header received
0923 20:09:16.846526 (+ 1970us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:09:16.846549 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:09:16.846967 (+ 418us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:09:16.847308 (+ 341us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:09:16.849188 (+ 1880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:09:16.850792 (+ 1604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:09:16.852956 (+ 2164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:09:16.853718 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:09:16.853856 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:09:16.858633 (+ 4777us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:09:16.858645 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:09:16.858665 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:09:16.858704 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:09:16.862188 (+ 3484us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:09:16.862982 (+ 794us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:09:16.862990 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:09:16.862997 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:09:16.863178 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:09:16.863552 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:09:16.863558 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:09:16.863562 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:09:16.864124 (+ 562us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:09:16.864432 (+ 308us) server_negotiation.cc:1035] Waiting for connection context
0923 20:09:16.864895 (+ 463us) server_negotiation.cc:299] Negotiation successful
0923 20:09:16.865339 (+ 444us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":584,"thread_start_us":374,"threads_started":1}
I20260923 20:09:16.868984 5356 raft_consensus.cc:348] T 00000000000000000000000000000000 P a02c794cff2d4fa89c47869d1107b5b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a02c794cff2d4fa89c47869d1107b5b5" member_type: VOTER last_known_addr { host: "127.1.170.190" port: 32911 } }
I20260923 20:09:16.869432 5356 raft_consensus.cc:374] T 00000000000000000000000000000000 P a02c794cff2d4fa89c47869d1107b5b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:09:16.869544 5356 raft_consensus.cc:729] T 00000000000000000000000000000000 P a02c794cff2d4fa89c47869d1107b5b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a02c794cff2d4fa89c47869d1107b5b5, State: Initialized, Role: FOLLOWER
I20260923 20:09:16.870105 5356 consensus_queue.cc:260] T 00000000000000000000000000000000 P a02c794cff2d4fa89c47869d1107b5b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a02c794cff2d4fa89c47869d1107b5b5" member_type: VOTER last_known_addr { host: "127.1.170.190" port: 32911 } }
I20260923 20:09:16.870322 5356 raft_consensus.cc:388] T 00000000000000000000000000000000 P a02c794cff2d4fa89c47869d1107b5b5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 20:09:16.870427 5356 raft_consensus.cc:482] T 00000000000000000000000000000000 P a02c794cff2d4fa89c47869d1107b5b5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 20:09:16.870589 5356 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a02c794cff2d4fa89c47869d1107b5b5 [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:09:16.873056 5356 raft_consensus.cc:504] T 00000000000000000000000000000000 P a02c794cff2d4fa89c47869d1107b5b5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a02c794cff2d4fa89c47869d1107b5b5" member_type: VOTER last_known_addr { host: "127.1.170.190" port: 32911 } }
I20260923 20:09:16.873576 5356 leader_election.cc:302] T 00000000000000000000000000000000 P a02c794cff2d4fa89c47869d1107b5b5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a02c794cff2d4fa89c47869d1107b5b5; no voters:
I20260923 20:09:16.874154 5356 leader_election.cc:288] T 00000000000000000000000000000000 P a02c794cff2d4fa89c47869d1107b5b5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 20:09:16.874322 5361 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a02c794cff2d4fa89c47869d1107b5b5 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 20:09:16.875198 5361 raft_consensus.cc:686] T 00000000000000000000000000000000 P a02c794cff2d4fa89c47869d1107b5b5 [term 1 LEADER]: Becoming Leader. State: Replica: a02c794cff2d4fa89c47869d1107b5b5, State: Running, Role: LEADER
I20260923 20:09:16.875808 5361 consensus_queue.cc:237] T 00000000000000000000000000000000 P a02c794cff2d4fa89c47869d1107b5b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a02c794cff2d4fa89c47869d1107b5b5" member_type: VOTER last_known_addr { host: "127.1.170.190" port: 32911 } }
I20260923 20:09:16.876482 5356 sys_catalog.cc:564] T 00000000000000000000000000000000 P a02c794cff2d4fa89c47869d1107b5b5 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 20:09:16.878492 5363 sys_catalog.cc:455] T 00000000000000000000000000000000 P a02c794cff2d4fa89c47869d1107b5b5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a02c794cff2d4fa89c47869d1107b5b5. Latest consensus state: current_term: 1 leader_uuid: "a02c794cff2d4fa89c47869d1107b5b5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a02c794cff2d4fa89c47869d1107b5b5" member_type: VOTER last_known_addr { host: "127.1.170.190" port: 32911 } } }
I20260923 20:09:16.878875 5363 sys_catalog.cc:458] T 00000000000000000000000000000000 P a02c794cff2d4fa89c47869d1107b5b5 [sys.catalog]: This master's current role is: LEADER
I20260923 20:09:16.880915 5362 sys_catalog.cc:455] T 00000000000000000000000000000000 P a02c794cff2d4fa89c47869d1107b5b5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a02c794cff2d4fa89c47869d1107b5b5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a02c794cff2d4fa89c47869d1107b5b5" member_type: VOTER last_known_addr { host: "127.1.170.190" port: 32911 } } }
I20260923 20:09:16.881425 5362 sys_catalog.cc:458] T 00000000000000000000000000000000 P a02c794cff2d4fa89c47869d1107b5b5 [sys.catalog]: This master's current role is: LEADER
I20260923 20:09:16.883020 5368 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 20:09:16.888427 5368 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 20:09:16.898414 5368 catalog_manager.cc:1347] Generated new cluster ID: b712865ab7dc4d60b9b2d5df6fba6027
I20260923 20:09:16.898605 5368 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 20:09:16.918288 5368 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 20:09:16.919852 5368 catalog_manager.cc:1504] Loading token signing keys...
I20260923 20:09:16.930163 5368 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a02c794cff2d4fa89c47869d1107b5b5: Generated new TSK 0
I20260923 20:09:16.931109 5368 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.170.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.170.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.170.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790194121577027-1706-0/krb5kdc/kudu_127.1.170.129.keytab.
Entry for principal kudu/127.1.170.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790194121577027-1706-0/krb5kdc/kudu_127.1.170.129.keytab.
I20260923 20:09:16.986943 1706 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqUuku6/build/asan/bin/kudu
/tmp/dist-test-taskqUuku6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790194121577027-1706-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790194121577027-1706-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790194121577027-1706-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790194121577027-1706-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.170.129:0
--local_ip_for_outbound_sockets=127.1.170.129
--webserver_interface=127.1.170.129
--webserver_port=0
--tserver_master_addrs=127.1.170.190:32911
--keytab_file=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790194121577027-1706-0/krb5kdc/kudu_127.1.170.129.keytab
--principal=kudu/127.1.170.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.170.148:43449
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790194121577027-1706-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790194121577027-1706-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790194121577027-1706-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:09:17.344048 5382 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:09:17.344360 5382 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:09:17.344502 5382 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:09:17.352124 5382 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:09:17.352260 5382 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:09:17.352449 5382 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.170.129
I20260923 20:09:17.363567 5382 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.170.148:43449
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790194121577027-1706-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790194121577027-1706-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790194121577027-1706-0/krb5kdc/kudu_127.1.170.129.keytab
--principal=kudu/127.1.170.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.170.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790194121577027-1706-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.170.129
--webserver_port=0
--tserver_master_addrs=127.1.170.190:32911
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.170.129
--log_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790194121577027-1706-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:09:17.364971 5382 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:09:17.366854 5382 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:09:17.382189 5390 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:09:17.382138 5388 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:09:17.382157 5387 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:09:17.383214 5382 server_base.cc:1034] running on GCE node
I20260923 20:09:17.383919 5382 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:09:17.385016 5382 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:09:17.386351 5382 hybrid_clock.cc:648] HybridClock initialized: now 1790194157386251 us; error 108 us; skew 500 ppm
I20260923 20:09:17.386713 5382 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:09:17 dist-test-slave-th4g krb5kdc[5276](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194157, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.170.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:09:17.390398 5382 init.cc:377] Logged in from keytab as kudu/127.1.170.129@KRBTEST.COM (short username kudu)
I20260923 20:09:17.392040 5382 webserver.cc:460] Webserver started at http://127.1.170.129:33409/ using document root <none> and password file <none>
I20260923 20:09:17.392794 5382 fs_manager.cc:359] Metadata directory not provided
I20260923 20:09:17.392946 5382 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:09:17.393332 5382 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:09:17.396602 5382 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790194121577027-1706-0/minicluster-data/ts-0/data/instance:
uuid: "1c20576a5b22412a8246a68421645f34"
format_stamp: "Formatted at 2026-09-23 20:09:17 on dist-test-slave-th4g"
server_key: "cb05f934bd442d0ed5c61243fd9f6b18"
server_key_iv: "252b79ab627ee8884387b0d03d2047a4"
server_key_version: "encryptionkey@0"
I20260923 20:09:17.397516 5382 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790194121577027-1706-0/minicluster-data/ts-0/wal/instance:
uuid: "1c20576a5b22412a8246a68421645f34"
format_stamp: "Formatted at 2026-09-23 20:09:17 on dist-test-slave-th4g"
server_key: "cb05f934bd442d0ed5c61243fd9f6b18"
server_key_iv: "252b79ab627ee8884387b0d03d2047a4"
server_key_version: "encryptionkey@0"
I20260923 20:09:17.403167 5382 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260923 20:09:17.407368 5397 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:09:17.409412 5382 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260923 20:09:17.409617 5382 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790194121577027-1706-0/minicluster-data/ts-0/data,/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790194121577027-1706-0/minicluster-data/ts-0/wal
uuid: "1c20576a5b22412a8246a68421645f34"
format_stamp: "Formatted at 2026-09-23 20:09:17 on dist-test-slave-th4g"
server_key: "cb05f934bd442d0ed5c61243fd9f6b18"
server_key_iv: "252b79ab627ee8884387b0d03d2047a4"
server_key_version: "encryptionkey@0"
I20260923 20:09:17.410250 5382 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790194121577027-1706-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790194121577027-1706-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790194121577027-1706-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:09:17.442811 5382 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:09:17.449231 5382 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:09:17.449601 5382 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:09:17.450870 5382 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 20:09:17.452383 5382 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 20:09:17.452495 5382 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 20:09:17.452638 5382 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 20:09:17.452713 5382 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 20:09:17.489929 5382 rpc_server.cc:307] RPC server started. Bound to: 127.1.170.129:44301
I20260923 20:09:17.490024 5509 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.170.129:44301 every 8 connection(s)
I20260923 20:09:17.491276 5382 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790194121577027-1706-0/minicluster-data/ts-0/data/info.pb
I20260923 20:09:17.499619 1706 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqUuku6/build/asan/bin/kudu as pid 5382
I20260923 20:09:17.499828 1706 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790194121577027-1706-0/minicluster-data/ts-0/wal/instance
I20260923 20:09:17.500478 1706 external_mini_cluster.cc:1428] Setting key e12fd31e976e0724ffec3869d7b54132
Sep 23 20:09:17 dist-test-slave-th4g krb5kdc[5276](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194157, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.170.129@KRBTEST.COM for kudu/127.1.170.190@KRBTEST.COM
I20260923 20:09:17.519091 5512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:09:17.494340 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.170.129:57827 (local address 127.1.170.190:32911)
0923 20:09:17.494830 (+ 490us) server_negotiation.cc:206] Beginning negotiation
0923 20:09:17.494837 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:09:17.496026 (+ 1189us) server_negotiation.cc:407] Connection header received
0923 20:09:17.499111 (+ 3085us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:09:17.499117 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:09:17.499230 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:09:17.499374 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:09:17.502208 (+ 2834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:09:17.503445 (+ 1237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:09:17.505101 (+ 1656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:09:17.505542 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:09:17.505686 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:09:17.510565 (+ 4879us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:09:17.510573 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:09:17.510576 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:09:17.510626 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:09:17.514372 (+ 3746us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:09:17.515632 (+ 1260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:09:17.515637 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:09:17.515639 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:09:17.515774 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:09:17.516444 (+ 670us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:09:17.516448 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:09:17.516450 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:09:17.516882 (+ 432us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:09:17.517007 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0923 20:09:17.518330 (+ 1323us) server_negotiation.cc:299] Negotiation successful
0923 20:09:17.518616 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":363,"thread_start_us":239,"threads_started":1}
I20260923 20:09:17.519971 5513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:09:17.494945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.190:32911 (local address 127.1.170.129:57827)
0923 20:09:17.495837 (+ 892us) negotiation.cc:107] Waiting for socket to connect
0923 20:09:17.495897 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0923 20:09:17.498668 (+ 2771us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:09:17.499673 (+ 1005us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:09:17.499688 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:09:17.500617 (+ 929us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:09:17.501909 (+ 1292us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:09:17.501941 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:09:17.503636 (+ 1695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:09:17.503641 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:09:17.504931 (+ 1290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:09:17.504944 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:09:17.505786 (+ 842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:09:17.505795 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:09:17.506092 (+ 297us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:09:17.506979 (+ 887us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:09:17.507015 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:09:17.509825 (+ 2810us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:09:17.514581 (+ 4756us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:09:17.514597 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:09:17.514623 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:09:17.515429 (+ 806us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:09:17.515929 (+ 500us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:09:17.515936 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:09:17.515943 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:09:17.516261 (+ 318us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:09:17.517023 (+ 762us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:09:17.517035 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:09:17.517551 (+ 516us) client_negotiation.cc:769] Sending connection context
0923 20:09:17.517971 (+ 420us) client_negotiation.cc:240] Negotiation successful
0923 20:09:17.518647 (+ 676us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":549,"thread_start_us":328,"threads_started":1}
I20260923 20:09:17.521915 5510 heartbeater.cc:347] Connected to a master server at 127.1.170.190:32911
I20260923 20:09:17.522426 5510 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.1.170.130@KRBTEST.COM; defaulting to no policy
I20260923 20:09:17.523808 5510 heartbeater.cc:511] Master 127.1.170.190:32911 requested a full tablet report, sending...
I20260923 20:09:17.526575 5321 ts_manager.cc:194] Registered new tserver with Master: 1c20576a5b22412a8246a68421645f34 (127.1.170.129:44301)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.170.130@KRBTEST.COM" created.
I20260923 20:09:17.528774 5321 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.170.129@KRBTEST.COM'} at 127.1.170.129:57827
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.170.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790194121577027-1706-0/krb5kdc/kudu_127.1.170.130.keytab.
Entry for principal kudu/127.1.170.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790194121577027-1706-0/krb5kdc/kudu_127.1.170.130.keytab.
I20260923 20:09:17.555320 1706 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqUuku6/build/asan/bin/kudu
/tmp/dist-test-taskqUuku6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790194121577027-1706-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790194121577027-1706-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790194121577027-1706-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790194121577027-1706-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.170.130:0
--local_ip_for_outbound_sockets=127.1.170.130
--webserver_interface=127.1.170.130
--webserver_port=0
--tserver_master_addrs=127.1.170.190:32911
--keytab_file=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790194121577027-1706-0/krb5kdc/kudu_127.1.170.130.keytab
--principal=kudu/127.1.170.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.170.148:43449
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790194121577027-1706-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790194121577027-1706-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790194121577027-1706-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:09:17.902930 5516 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:09:17.903295 5516 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:09:17.903462 5516 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:09:17.911001 5516 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:09:17.911180 5516 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:09:17.911391 5516 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.170.130
I20260923 20:09:17.921595 5516 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.170.148:43449
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790194121577027-1706-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790194121577027-1706-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790194121577027-1706-0/krb5kdc/kudu_127.1.170.130.keytab
--principal=kudu/127.1.170.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.170.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790194121577027-1706-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.170.130
--webserver_port=0
--tserver_master_addrs=127.1.170.190:32911
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.170.130
--log_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790194121577027-1706-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:09:17.922991 5516 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:09:17.924332 5516 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:09:17.940002 5522 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:09:17.940069 5521 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:09:17.940284 5524 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:09:17.941272 5516 server_base.cc:1034] running on GCE node
I20260923 20:09:17.942374 5516 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:09:17.943926 5516 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:09:17.945204 5516 hybrid_clock.cc:648] HybridClock initialized: now 1790194157945147 us; error 52 us; skew 500 ppm
I20260923 20:09:17.945578 5516 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:09:17 dist-test-slave-th4g krb5kdc[5276](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194157, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.170.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:09:17.949535 5516 init.cc:377] Logged in from keytab as kudu/127.1.170.130@KRBTEST.COM (short username kudu)
I20260923 20:09:17.951228 5516 webserver.cc:460] Webserver started at http://127.1.170.130:36085/ using document root <none> and password file <none>
I20260923 20:09:17.951921 5516 fs_manager.cc:359] Metadata directory not provided
I20260923 20:09:17.952028 5516 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:09:17.952286 5516 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:09:17.956403 5516 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790194121577027-1706-0/minicluster-data/ts-1/data/instance:
uuid: "8a972f24727349d99e09bc4e6d2a8cf7"
format_stamp: "Formatted at 2026-09-23 20:09:17 on dist-test-slave-th4g"
server_key: "bf52f6e005345e08605b9a7e5cdb5966"
server_key_iv: "a57004bc234f1e9c731eb61c1ab72c20"
server_key_version: "encryptionkey@0"
I20260923 20:09:17.957523 5516 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790194121577027-1706-0/minicluster-data/ts-1/wal/instance:
uuid: "8a972f24727349d99e09bc4e6d2a8cf7"
format_stamp: "Formatted at 2026-09-23 20:09:17 on dist-test-slave-th4g"
server_key: "bf52f6e005345e08605b9a7e5cdb5966"
server_key_iv: "a57004bc234f1e9c731eb61c1ab72c20"
server_key_version: "encryptionkey@0"
I20260923 20:09:17.963265 5516 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260923 20:09:17.967278 5531 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:09:17.968725 5516 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260923 20:09:17.969019 5516 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790194121577027-1706-0/minicluster-data/ts-1/data,/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790194121577027-1706-0/minicluster-data/ts-1/wal
uuid: "8a972f24727349d99e09bc4e6d2a8cf7"
format_stamp: "Formatted at 2026-09-23 20:09:17 on dist-test-slave-th4g"
server_key: "bf52f6e005345e08605b9a7e5cdb5966"
server_key_iv: "a57004bc234f1e9c731eb61c1ab72c20"
server_key_version: "encryptionkey@0"
I20260923 20:09:17.969252 5516 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790194121577027-1706-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790194121577027-1706-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790194121577027-1706-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:09:17.987493 5516 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:09:17.991905 5516 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:09:17.992200 5516 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:09:17.993183 5516 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 20:09:17.994623 5516 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 20:09:17.994696 5516 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 20:09:17.994830 5516 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 20:09:17.994870 5516 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 20:09:18.030726 5516 rpc_server.cc:307] RPC server started. Bound to: 127.1.170.130:33775
I20260923 20:09:18.030802 5643 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.170.130:33775 every 8 connection(s)
I20260923 20:09:18.032091 5516 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790194121577027-1706-0/minicluster-data/ts-1/data/info.pb
I20260923 20:09:18.036712 1706 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqUuku6/build/asan/bin/kudu as pid 5516
I20260923 20:09:18.036937 1706 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790194121577027-1706-0/minicluster-data/ts-1/wal/instance
I20260923 20:09:18.038002 1706 external_mini_cluster.cc:1428] Setting key 9578dcca2f1e74224a71b05476f1734c
Sep 23 20:09:18 dist-test-slave-th4g krb5kdc[5276](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194157, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.170.130@KRBTEST.COM for kudu/127.1.170.190@KRBTEST.COM
I20260923 20:09:18.059594 5646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:09:18.035010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.170.130:45209 (local address 127.1.170.190:32911)
0923 20:09:18.035462 (+ 452us) server_negotiation.cc:206] Beginning negotiation
0923 20:09:18.035467 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 20:09:18.037347 (+ 1880us) server_negotiation.cc:407] Connection header received
0923 20:09:18.039866 (+ 2519us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:09:18.039872 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:09:18.039970 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:09:18.040083 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:09:18.042122 (+ 2039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:09:18.042970 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:09:18.045546 (+ 2576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:09:18.046250 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:09:18.046391 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:09:18.050979 (+ 4588us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:09:18.050990 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:09:18.050998 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:09:18.051048 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:09:18.054830 (+ 3782us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:09:18.056356 (+ 1526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:09:18.056366 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:09:18.056372 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:09:18.056557 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:09:18.057159 (+ 602us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:09:18.057166 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:09:18.057172 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:09:18.057630 (+ 458us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:09:18.057826 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0923 20:09:18.058896 (+ 1070us) server_negotiation.cc:299] Negotiation successful
0923 20:09:18.059153 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":353,"thread_start_us":248,"threads_started":1}
WARNING: no policy specified for kudu/127.1.170.131@KRBTEST.COM; defaulting to no policy
I20260923 20:09:18.061082 5647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:09:18.035525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.190:32911 (local address 127.1.170.130:45209)
0923 20:09:18.036957 (+ 1432us) negotiation.cc:107] Waiting for socket to connect
0923 20:09:18.037138 (+ 181us) client_negotiation.cc:174] Beginning negotiation
0923 20:09:18.039590 (+ 2452us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:09:18.040326 (+ 736us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:09:18.040335 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:09:18.041026 (+ 691us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:09:18.041869 (+ 843us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:09:18.041933 (+ 64us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:09:18.043164 (+ 1231us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:09:18.043172 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:09:18.045365 (+ 2193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:09:18.045384 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:09:18.046515 (+ 1131us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:09:18.046525 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:09:18.046846 (+ 321us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:09:18.047693 (+ 847us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:09:18.047721 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:09:18.050688 (+ 2967us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:09:18.055106 (+ 4418us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:09:18.055119 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:09:18.055141 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:09:18.056075 (+ 934us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:09:18.056723 (+ 648us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:09:18.056727 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:09:18.056729 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:09:18.056999 (+ 270us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:09:18.057861 (+ 862us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:09:18.057871 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:09:18.058460 (+ 589us) client_negotiation.cc:769] Sending connection context
0923 20:09:18.058736 (+ 276us) client_negotiation.cc:240] Negotiation successful
0923 20:09:18.059354 (+ 618us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":916,"thread_start_us":515,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.170.131@KRBTEST.COM" created.
I20260923 20:09:18.063521 5644 heartbeater.cc:347] Connected to a master server at 127.1.170.190:32911
I20260923 20:09:18.064081 5644 heartbeater.cc:464] Registering TS with master...
I20260923 20:09:18.065163 5644 heartbeater.cc:511] Master 127.1.170.190:32911 requested a full tablet report, sending...
I20260923 20:09:18.067103 5321 ts_manager.cc:194] Registered new tserver with Master: 8a972f24727349d99e09bc4e6d2a8cf7 (127.1.170.130:33775)
I20260923 20:09:18.068404 5321 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.170.130@KRBTEST.COM'} at 127.1.170.130:45209
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.170.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790194121577027-1706-0/krb5kdc/kudu_127.1.170.131.keytab.
Entry for principal kudu/127.1.170.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790194121577027-1706-0/krb5kdc/kudu_127.1.170.131.keytab.
I20260923 20:09:18.089509 1706 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqUuku6/build/asan/bin/kudu
/tmp/dist-test-taskqUuku6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790194121577027-1706-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790194121577027-1706-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790194121577027-1706-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790194121577027-1706-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.170.131:0
--local_ip_for_outbound_sockets=127.1.170.131
--webserver_interface=127.1.170.131
--webserver_port=0
--tserver_master_addrs=127.1.170.190:32911
--keytab_file=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790194121577027-1706-0/krb5kdc/kudu_127.1.170.131.keytab
--principal=kudu/127.1.170.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.170.148:43449
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790194121577027-1706-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790194121577027-1706-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790194121577027-1706-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:09:18.416882 5650 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:09:18.417143 5650 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:09:18.417322 5650 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:09:18.424575 5650 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:09:18.424686 5650 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:09:18.424860 5650 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.170.131
I20260923 20:09:18.434615 5650 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.170.148:43449
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790194121577027-1706-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790194121577027-1706-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790194121577027-1706-0/krb5kdc/kudu_127.1.170.131.keytab
--principal=kudu/127.1.170.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.170.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790194121577027-1706-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.170.131
--webserver_port=0
--tserver_master_addrs=127.1.170.190:32911
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.170.131
--log_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790194121577027-1706-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:09:18.435904 5650 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:09:18.437189 5650 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:09:18.450340 5655 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:09:18.450323 5656 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:09:18.450873 5658 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:09:18.451288 5650 server_base.cc:1034] running on GCE node
I20260923 20:09:18.451864 5650 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:09:18.452833 5650 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:09:18.454012 5650 hybrid_clock.cc:648] HybridClock initialized: now 1790194158453972 us; error 40 us; skew 500 ppm
I20260923 20:09:18.454286 5650 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:09:18 dist-test-slave-th4g krb5kdc[5276](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194158, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.170.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:09:18.457425 5650 init.cc:377] Logged in from keytab as kudu/127.1.170.131@KRBTEST.COM (short username kudu)
I20260923 20:09:18.458892 5650 webserver.cc:460] Webserver started at http://127.1.170.131:33017/ using document root <none> and password file <none>
I20260923 20:09:18.459571 5650 fs_manager.cc:359] Metadata directory not provided
I20260923 20:09:18.459697 5650 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:09:18.459996 5650 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:09:18.462913 5650 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790194121577027-1706-0/minicluster-data/ts-2/data/instance:
uuid: "e8ecb41d41fb484c8e96b933d3fdee49"
format_stamp: "Formatted at 2026-09-23 20:09:18 on dist-test-slave-th4g"
server_key: "1bd7b96f3ad9c0e048e851c29d367840"
server_key_iv: "699d92b207817616517d9a51cb963670"
server_key_version: "encryptionkey@0"
I20260923 20:09:18.463675 5650 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790194121577027-1706-0/minicluster-data/ts-2/wal/instance:
uuid: "e8ecb41d41fb484c8e96b933d3fdee49"
format_stamp: "Formatted at 2026-09-23 20:09:18 on dist-test-slave-th4g"
server_key: "1bd7b96f3ad9c0e048e851c29d367840"
server_key_iv: "699d92b207817616517d9a51cb963670"
server_key_version: "encryptionkey@0"
I20260923 20:09:18.469174 5650 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.005s
I20260923 20:09:18.472774 5665 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:09:18.474203 5650 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260923 20:09:18.474366 5650 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790194121577027-1706-0/minicluster-data/ts-2/data,/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790194121577027-1706-0/minicluster-data/ts-2/wal
uuid: "e8ecb41d41fb484c8e96b933d3fdee49"
format_stamp: "Formatted at 2026-09-23 20:09:18 on dist-test-slave-th4g"
server_key: "1bd7b96f3ad9c0e048e851c29d367840"
server_key_iv: "699d92b207817616517d9a51cb963670"
server_key_version: "encryptionkey@0"
I20260923 20:09:18.474567 5650 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790194121577027-1706-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790194121577027-1706-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790194121577027-1706-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:09:18.495563 5650 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:09:18.500026 5650 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:09:18.500321 5650 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:09:18.501283 5650 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 20:09:18.502701 5650 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 20:09:18.502796 5650 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 20:09:18.502949 5650 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 20:09:18.503010 5650 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 20:09:18.532471 5510 heartbeater.cc:503] Master 127.1.170.190:32911 was elected leader, sending a full tablet report...
I20260923 20:09:18.536273 5650 rpc_server.cc:307] RPC server started. Bound to: 127.1.170.131:34987
I20260923 20:09:18.536373 5777 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.170.131:34987 every 8 connection(s)
I20260923 20:09:18.537537 5650 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790194121577027-1706-0/minicluster-data/ts-2/data/info.pb
I20260923 20:09:18.538729 1706 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqUuku6/build/asan/bin/kudu as pid 5650
I20260923 20:09:18.538921 1706 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790194121577027-1706-0/minicluster-data/ts-2/wal/instance
I20260923 20:09:18.539467 1706 external_mini_cluster.cc:1428] Setting key 31fd934510f3eaca62c27be8b71c526a
Sep 23 20:09:18 dist-test-slave-th4g krb5kdc[5276](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194158, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.170.131@KRBTEST.COM for kudu/127.1.170.190@KRBTEST.COM
I20260923 20:09:18.558195 5646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:09:18.540533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.170.131:40221 (local address 127.1.170.190:32911)
0923 20:09:18.540716 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0923 20:09:18.540720 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 20:09:18.542327 (+ 1607us) server_negotiation.cc:407] Connection header received
0923 20:09:18.544242 (+ 1915us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:09:18.544246 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:09:18.544323 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:09:18.544405 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:09:18.545797 (+ 1392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:09:18.546595 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:09:18.548280 (+ 1685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:09:18.548648 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:09:18.548752 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:09:18.552264 (+ 3512us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:09:18.552269 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:09:18.552273 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:09:18.552304 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:09:18.555485 (+ 3181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:09:18.556183 (+ 698us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:09:18.556187 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:09:18.556189 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:09:18.556313 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:09:18.556677 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:09:18.556681 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:09:18.556683 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:09:18.556942 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:09:18.557049 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0923 20:09:18.557729 (+ 680us) server_negotiation.cc:299] Negotiation successful
0923 20:09:18.557960 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260923 20:09:18.559257 5780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:09:18.541057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.190:32911 (local address 127.1.170.131:40221)
0923 20:09:18.542146 (+ 1089us) negotiation.cc:107] Waiting for socket to connect
0923 20:09:18.542205 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0923 20:09:18.543997 (+ 1792us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:09:18.544595 (+ 598us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:09:18.544603 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:09:18.545088 (+ 485us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:09:18.545643 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:09:18.545663 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:09:18.546755 (+ 1092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:09:18.546759 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:09:18.548126 (+ 1367us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:09:18.548138 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:09:18.548834 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:09:18.548839 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:09:18.549034 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:09:18.549747 (+ 713us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:09:18.549769 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:09:18.552036 (+ 2267us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:09:18.555617 (+ 3581us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:09:18.555624 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:09:18.555638 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:09:18.556076 (+ 438us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:09:18.556408 (+ 332us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:09:18.556412 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:09:18.556414 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:09:18.556590 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:09:18.557039 (+ 449us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:09:18.557046 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:09:18.557369 (+ 323us) client_negotiation.cc:769] Sending connection context
0923 20:09:18.557603 (+ 234us) client_negotiation.cc:240] Negotiation successful
0923 20:09:18.558061 (+ 458us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":780,"thread_start_us":543,"threads_started":1}
I20260923 20:09:18.560628 5778 heartbeater.cc:347] Connected to a master server at 127.1.170.190:32911
I20260923 20:09:18.560940 5778 heartbeater.cc:464] Registering TS with master...
I20260923 20:09:18.561619 5778 heartbeater.cc:511] Master 127.1.170.190:32911 requested a full tablet report, sending...
I20260923 20:09:18.563032 5321 ts_manager.cc:194] Registered new tserver with Master: e8ecb41d41fb484c8e96b933d3fdee49 (127.1.170.131:34987)
I20260923 20:09:18.563758 5321 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.170.131@KRBTEST.COM'} at 127.1.170.131:40221
I20260923 20:09:18.567471 1706 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 23 20:09:18 dist-test-slave-th4g krb5kdc[5276](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194158, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 23 20:09:18 dist-test-slave-th4g krb5kdc[5276](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194158, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.170.129@KRBTEST.COM
I20260923 20:09:18.614985 5785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:09:18.599952 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46448 (local address 127.1.170.129:44301)
0923 20:09:18.600654 (+ 702us) server_negotiation.cc:206] Beginning negotiation
0923 20:09:18.600661 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:09:18.600685 (+ 24us) server_negotiation.cc:407] Connection header received
0923 20:09:18.600747 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:09:18.600754 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:09:18.600974 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:09:18.601147 (+ 173us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:09:18.602504 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:09:18.603477 (+ 973us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:09:18.604697 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:09:18.605244 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:09:18.605378 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:09:18.609736 (+ 4358us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:09:18.609747 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:09:18.609761 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:09:18.609793 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:09:18.612455 (+ 2662us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:09:18.613097 (+ 642us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:09:18.613116 (+ 19us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:09:18.613121 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:09:18.613261 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:09:18.613581 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:09:18.613584 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:09:18.613586 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:09:18.613950 (+ 364us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:09:18.614176 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0923 20:09:18.614411 (+ 235us) server_negotiation.cc:299] Negotiation successful
0923 20:09:18.614611 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":543,"thread_start_us":353,"threads_started":1}
Sep 23 20:09:18 dist-test-slave-th4g krb5kdc[5276](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194158, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.170.190@KRBTEST.COM
I20260923 20:09:18.631646 5646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:09:18.619491 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38436 (local address 127.1.170.190:32911)
0923 20:09:18.619698 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0923 20:09:18.619703 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 20:09:18.619770 (+ 67us) server_negotiation.cc:407] Connection header received
0923 20:09:18.619929 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:09:18.619933 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:09:18.620007 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:09:18.620086 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:09:18.621212 (+ 1126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:09:18.621968 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:09:18.623090 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:09:18.623479 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:09:18.623570 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:09:18.627039 (+ 3469us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:09:18.627043 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:09:18.627046 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:09:18.627075 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:09:18.629485 (+ 2410us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:09:18.630079 (+ 594us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:09:18.630083 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:09:18.630085 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:09:18.630202 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:09:18.630485 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:09:18.630488 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:09:18.630490 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:09:18.630776 (+ 286us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:09:18.630954 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0923 20:09:18.631156 (+ 202us) server_negotiation.cc:299] Negotiation successful
0923 20:09:18.631300 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
W20260923 20:09:18.632185 5321 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38436
I20260923 20:09:18.633055 1706 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskqUuku6/build/asan/bin/kudu pid 5382
I20260923 20:09:18.645181 5785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:09:18.634005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46456 (local address 127.1.170.129:44301)
0923 20:09:18.634190 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0923 20:09:18.634196 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:09:18.634312 (+ 116us) server_negotiation.cc:407] Connection header received
0923 20:09:18.634482 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:09:18.634487 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:09:18.634586 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:09:18.634687 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:09:18.635810 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:09:18.636487 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:09:18.637647 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:09:18.638135 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:09:18.638269 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:09:18.640359 (+ 2090us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:09:18.640365 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:09:18.640370 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:09:18.640406 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:09:18.642803 (+ 2397us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:09:18.643393 (+ 590us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:09:18.643401 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:09:18.643404 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:09:18.643520 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:09:18.643870 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:09:18.643877 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:09:18.643880 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:09:18.644126 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:09:18.644375 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0923 20:09:18.644663 (+ 288us) server_negotiation.cc:299] Negotiation successful
0923 20:09:18.644890 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260923 20:09:18.645959 5505 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:09:18.727119 1706 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqUuku6/build/asan/bin/kudu with pid 5382
I20260923 20:09:18.743108 1706 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskqUuku6/build/asan/bin/kudu pid 5516
Sep 23 20:09:18 dist-test-slave-th4g krb5kdc[5276](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194158, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.170.130@KRBTEST.COM
I20260923 20:09:18.758106 5790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:09:18.744013 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53136 (local address 127.1.170.130:33775)
0923 20:09:18.744511 (+ 498us) server_negotiation.cc:206] Beginning negotiation
0923 20:09:18.744518 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:09:18.744543 (+ 25us) server_negotiation.cc:407] Connection header received
0923 20:09:18.744604 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:09:18.744611 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:09:18.744811 (+ 200us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:09:18.744954 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:09:18.746115 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:09:18.747087 (+ 972us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:09:18.748228 (+ 1141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:09:18.748736 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:09:18.748837 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:09:18.752644 (+ 3807us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:09:18.752654 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:09:18.752667 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:09:18.752696 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:09:18.755691 (+ 2995us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:09:18.756261 (+ 570us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:09:18.756267 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:09:18.756270 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:09:18.756383 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:09:18.756665 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:09:18.756668 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:09:18.756670 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:09:18.757002 (+ 332us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:09:18.757233 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0923 20:09:18.757491 (+ 258us) server_negotiation.cc:299] Negotiation successful
0923 20:09:18.757667 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":379,"thread_start_us":256,"threads_started":1}
I20260923 20:09:18.759570 5639 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:09:18.841837 1706 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqUuku6/build/asan/bin/kudu with pid 5516
I20260923 20:09:18.857331 1706 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskqUuku6/build/asan/bin/kudu pid 5650
Sep 23 20:09:18 dist-test-slave-th4g krb5kdc[5276](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194158, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.170.131@KRBTEST.COM
I20260923 20:09:18.874163 5792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:09:18.858295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43898 (local address 127.1.170.131:34987)
0923 20:09:18.858919 (+ 624us) server_negotiation.cc:206] Beginning negotiation
0923 20:09:18.858926 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:09:18.858954 (+ 28us) server_negotiation.cc:407] Connection header received
0923 20:09:18.859012 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:09:18.859018 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:09:18.859276 (+ 258us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:09:18.859398 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:09:18.860610 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:09:18.861692 (+ 1082us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:09:18.863136 (+ 1444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:09:18.863699 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:09:18.863805 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:09:18.868253 (+ 4448us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:09:18.868267 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:09:18.868295 (+ 28us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:09:18.868350 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:09:18.871453 (+ 3103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:09:18.872171 (+ 718us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:09:18.872180 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:09:18.872184 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:09:18.872310 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:09:18.872658 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:09:18.872662 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:09:18.872664 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:09:18.873034 (+ 370us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:09:18.873293 (+ 259us) server_negotiation.cc:1035] Waiting for connection context
0923 20:09:18.873550 (+ 257us) server_negotiation.cc:299] Negotiation successful
0923 20:09:18.873768 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":477,"thread_start_us":313,"threads_started":1}
I20260923 20:09:18.875537 5773 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:09:18.959652 1706 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqUuku6/build/asan/bin/kudu with pid 5650
I20260923 20:09:18.975648 1706 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskqUuku6/build/asan/bin/kudu pid 5289
I20260923 20:09:18.977169 5351 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:09:19.078804 1706 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqUuku6/build/asan/bin/kudu with pid 5289
2026-09-23T20:09:19Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (4076 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790194121577027-1706-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 20:09:19 dist-test-slave-th4g krb5kdc[5797](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 20:09:19 dist-test-slave-th4g krb5kdc[5797](info): set up 2 sockets
Sep 23 20:09:19 dist-test-slave-th4g krb5kdc[5797](info): commencing operation
krb5kdc: starting...
W20260923 20:09:20.175535 1706 mini_kdc.cc:121] Time spent starting KDC: real 1.055s user 0.003s sys 0.015s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790194121577027-1706-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790194121577027-1706-0/krb5kdc/test-user.keytab.
Sep 23 20:09:20 dist-test-slave-th4g krb5kdc[5797](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194160, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T20:09:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T20:09:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.170.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.170.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.170.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790194121577027-1706-0/krb5kdc/kudu_127.1.170.190.keytab.
Entry for principal kudu/127.1.170.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790194121577027-1706-0/krb5kdc/kudu_127.1.170.190.keytab.
I20260923 20:09:20.378353 1706 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqUuku6/build/asan/bin/kudu
/tmp/dist-test-taskqUuku6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790194121577027-1706-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790194121577027-1706-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790194121577027-1706-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790194121577027-1706-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.170.190:45463
--webserver_interface=127.1.170.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790194121577027-1706-0/krb5kdc/kudu_127.1.170.190.keytab
--principal=kudu/127.1.170.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.170.148:45765
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.170.190:45463
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790194121577027-1706-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790194121577027-1706-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790194121577027-1706-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:09:20.715610 5810 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:09:20.715941 5810 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:09:20.716125 5810 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:09:20.724370 5810 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 20:09:20.724501 5810 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:09:20.724584 5810 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:09:20.724668 5810 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 20:09:20.724742 5810 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 20:09:20.735011 5810 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.170.148:45765
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790194121577027-1706-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790194121577027-1706-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.170.190:45463
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790194121577027-1706-0/krb5kdc/kudu_127.1.170.190.keytab
--principal=kudu/127.1.170.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.170.190:45463
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790194121577027-1706-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.170.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790194121577027-1706-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:09:20.736402 5810 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:09:20.737744 5810 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:09:20.749784 5815 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:09:20.749758 5816 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:09:20.749718 5818 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:09:20.750597 5810 server_base.cc:1034] running on GCE node
I20260923 20:09:20.751386 5810 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:09:20.752843 5810 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:09:20.754065 5810 hybrid_clock.cc:648] HybridClock initialized: now 1790194160754009 us; error 43 us; skew 500 ppm
I20260923 20:09:20.754495 5810 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:09:20 dist-test-slave-th4g krb5kdc[5797](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194160, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.170.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:09:20.757694 5810 init.cc:377] Logged in from keytab as kudu/127.1.170.190@KRBTEST.COM (short username kudu)
I20260923 20:09:20.758980 5810 fs_manager.cc:359] Metadata directory not provided
I20260923 20:09:20.759111 5810 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:09:20.759438 5810 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:09:20.762542 5810 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790194121577027-1706-0/minicluster-data/master-0/data/instance:
uuid: "ea263cafa53b48238eb244aa75235503"
format_stamp: "Formatted at 2026-09-23 20:09:20 on dist-test-slave-th4g"
server_key: "162e64880e84662b04ef13f03ef76a2f"
server_key_iv: "4b235d7e61c1397c2c1426ede4320c4d"
server_key_version: "encryptionkey@0"
I20260923 20:09:20.763360 5810 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790194121577027-1706-0/minicluster-data/master-0/wal/instance:
uuid: "ea263cafa53b48238eb244aa75235503"
format_stamp: "Formatted at 2026-09-23 20:09:20 on dist-test-slave-th4g"
server_key: "162e64880e84662b04ef13f03ef76a2f"
server_key_iv: "4b235d7e61c1397c2c1426ede4320c4d"
server_key_version: "encryptionkey@0"
I20260923 20:09:20.768631 5810 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260923 20:09:20.772270 5824 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:09:20.773764 5810 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260923 20:09:20.774020 5810 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790194121577027-1706-0/minicluster-data/master-0/data,/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790194121577027-1706-0/minicluster-data/master-0/wal
uuid: "ea263cafa53b48238eb244aa75235503"
format_stamp: "Formatted at 2026-09-23 20:09:20 on dist-test-slave-th4g"
server_key: "162e64880e84662b04ef13f03ef76a2f"
server_key_iv: "4b235d7e61c1397c2c1426ede4320c4d"
server_key_version: "encryptionkey@0"
I20260923 20:09:20.774266 5810 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790194121577027-1706-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790194121577027-1706-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790194121577027-1706-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:09:20.793668 5810 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:09:20.798377 5810 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:09:20.798674 5810 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:09:20.819100 5875 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.170.190:45463 every 8 connection(s)
I20260923 20:09:20.819178 5810 rpc_server.cc:307] RPC server started. Bound to: 127.1.170.190:45463
I20260923 20:09:20.820103 5810 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790194121577027-1706-0/minicluster-data/master-0/data/info.pb
I20260923 20:09:20.823992 5876 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:09:20.829452 1706 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqUuku6/build/asan/bin/kudu as pid 5810
I20260923 20:09:20.829662 1706 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790194121577027-1706-0/minicluster-data/master-0/wal/instance
I20260923 20:09:20.830502 1706 external_mini_cluster.cc:1428] Setting key 3c044ea224ae4c012ec539da14dd4005
I20260923 20:09:20.835958 5876 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ea263cafa53b48238eb244aa75235503: Bootstrap starting.
I20260923 20:09:20.839769 5876 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ea263cafa53b48238eb244aa75235503: Neither blocks nor log segments found. Creating new log.
I20260923 20:09:20.841085 5876 log.cc:824] T 00000000000000000000000000000000 P ea263cafa53b48238eb244aa75235503: Log is configured to *not* fsync() on all Append() calls
I20260923 20:09:20.844928 5876 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ea263cafa53b48238eb244aa75235503: No bootstrap required, opened a new log
Sep 23 20:09:20 dist-test-slave-th4g krb5kdc[5797](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194160, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.170.190@KRBTEST.COM
I20260923 20:09:20.853302 5876 raft_consensus.cc:348] T 00000000000000000000000000000000 P ea263cafa53b48238eb244aa75235503 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea263cafa53b48238eb244aa75235503" member_type: VOTER last_known_addr { host: "127.1.170.190" port: 45463 } }
I20260923 20:09:20.853704 5876 raft_consensus.cc:374] T 00000000000000000000000000000000 P ea263cafa53b48238eb244aa75235503 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:09:20.853817 5876 raft_consensus.cc:729] T 00000000000000000000000000000000 P ea263cafa53b48238eb244aa75235503 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea263cafa53b48238eb244aa75235503, State: Initialized, Role: FOLLOWER
I20260923 20:09:20.854516 5876 consensus_queue.cc:260] T 00000000000000000000000000000000 P ea263cafa53b48238eb244aa75235503 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea263cafa53b48238eb244aa75235503" member_type: VOTER last_known_addr { host: "127.1.170.190" port: 45463 } }
I20260923 20:09:20.854760 5876 raft_consensus.cc:388] T 00000000000000000000000000000000 P ea263cafa53b48238eb244aa75235503 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 20:09:20.854907 5876 raft_consensus.cc:482] T 00000000000000000000000000000000 P ea263cafa53b48238eb244aa75235503 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 20:09:20.855094 5876 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ea263cafa53b48238eb244aa75235503 [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:09:20.855501 5879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:09:20.832695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60912 (local address 127.1.170.190:45463)
0923 20:09:20.833685 (+ 990us) server_negotiation.cc:206] Beginning negotiation
0923 20:09:20.833697 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0923 20:09:20.833739 (+ 42us) server_negotiation.cc:407] Connection header received
0923 20:09:20.835518 (+ 1779us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:09:20.835543 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:09:20.835956 (+ 413us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:09:20.836344 (+ 388us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:09:20.838256 (+ 1912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:09:20.839692 (+ 1436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:09:20.841501 (+ 1809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:09:20.842778 (+ 1277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:09:20.842944 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:09:20.847700 (+ 4756us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:09:20.847712 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:09:20.847732 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:09:20.847774 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:09:20.851025 (+ 3251us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:09:20.851779 (+ 754us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:09:20.851786 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:09:20.851793 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:09:20.851979 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:09:20.852366 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:09:20.852370 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:09:20.852378 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:09:20.852956 (+ 578us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:09:20.853277 (+ 321us) server_negotiation.cc:1035] Waiting for connection context
0923 20:09:20.853726 (+ 449us) server_negotiation.cc:299] Negotiation successful
0923 20:09:20.854331 (+ 605us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":775,"thread_start_us":523,"threads_started":1}
I20260923 20:09:20.858124 5876 raft_consensus.cc:504] T 00000000000000000000000000000000 P ea263cafa53b48238eb244aa75235503 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea263cafa53b48238eb244aa75235503" member_type: VOTER last_known_addr { host: "127.1.170.190" port: 45463 } }
I20260923 20:09:20.858716 5876 leader_election.cc:302] T 00000000000000000000000000000000 P ea263cafa53b48238eb244aa75235503 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ea263cafa53b48238eb244aa75235503; no voters:
I20260923 20:09:20.859386 5876 leader_election.cc:288] T 00000000000000000000000000000000 P ea263cafa53b48238eb244aa75235503 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 20:09:20.859851 5881 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ea263cafa53b48238eb244aa75235503 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 20:09:20.860780 5881 raft_consensus.cc:686] T 00000000000000000000000000000000 P ea263cafa53b48238eb244aa75235503 [term 1 LEADER]: Becoming Leader. State: Replica: ea263cafa53b48238eb244aa75235503, State: Running, Role: LEADER
I20260923 20:09:20.861399 5881 consensus_queue.cc:237] T 00000000000000000000000000000000 P ea263cafa53b48238eb244aa75235503 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea263cafa53b48238eb244aa75235503" member_type: VOTER last_known_addr { host: "127.1.170.190" port: 45463 } }
I20260923 20:09:20.861645 5876 sys_catalog.cc:564] T 00000000000000000000000000000000 P ea263cafa53b48238eb244aa75235503 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 20:09:20.869742 5883 sys_catalog.cc:455] T 00000000000000000000000000000000 P ea263cafa53b48238eb244aa75235503 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ea263cafa53b48238eb244aa75235503. Latest consensus state: current_term: 1 leader_uuid: "ea263cafa53b48238eb244aa75235503" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea263cafa53b48238eb244aa75235503" member_type: VOTER last_known_addr { host: "127.1.170.190" port: 45463 } } }
I20260923 20:09:20.870437 5883 sys_catalog.cc:458] T 00000000000000000000000000000000 P ea263cafa53b48238eb244aa75235503 [sys.catalog]: This master's current role is: LEADER
I20260923 20:09:20.871441 5882 sys_catalog.cc:455] T 00000000000000000000000000000000 P ea263cafa53b48238eb244aa75235503 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ea263cafa53b48238eb244aa75235503" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea263cafa53b48238eb244aa75235503" member_type: VOTER last_known_addr { host: "127.1.170.190" port: 45463 } } }
I20260923 20:09:20.871816 5882 sys_catalog.cc:458] T 00000000000000000000000000000000 P ea263cafa53b48238eb244aa75235503 [sys.catalog]: This master's current role is: LEADER
W20260923 20:09:20.879628 5897 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ea263cafa53b48238eb244aa75235503: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260923 20:09:20.879910 5897 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260923 20:09:20.880270 5891 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 20:09:20.882052 5891 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 20:09:20.890805 5891 catalog_manager.cc:1347] Generated new cluster ID: 08002d80bb5f4e04a5afd5627057d651
I20260923 20:09:20.890959 5891 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 20:09:20.909817 5891 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 20:09:20.911243 5891 catalog_manager.cc:1504] Loading token signing keys...
I20260923 20:09:20.922237 5891 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ea263cafa53b48238eb244aa75235503: Generated new TSK 0
I20260923 20:09:20.923146 5891 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.170.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.170.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.170.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790194121577027-1706-0/krb5kdc/kudu_127.1.170.129.keytab.
Entry for principal kudu/127.1.170.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790194121577027-1706-0/krb5kdc/kudu_127.1.170.129.keytab.
I20260923 20:09:20.974342 1706 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqUuku6/build/asan/bin/kudu
/tmp/dist-test-taskqUuku6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790194121577027-1706-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790194121577027-1706-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790194121577027-1706-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790194121577027-1706-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.170.129:0
--local_ip_for_outbound_sockets=127.1.170.129
--webserver_interface=127.1.170.129
--webserver_port=0
--tserver_master_addrs=127.1.170.190:45463
--keytab_file=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790194121577027-1706-0/krb5kdc/kudu_127.1.170.129.keytab
--principal=kudu/127.1.170.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.170.148:45765
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790194121577027-1706-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790194121577027-1706-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790194121577027-1706-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:09:21.352380 5902 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:09:21.352681 5902 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:09:21.352819 5902 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:09:21.360734 5902 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:09:21.360863 5902 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:09:21.361037 5902 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.170.129
I20260923 20:09:21.371536 5902 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.170.148:45765
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790194121577027-1706-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790194121577027-1706-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790194121577027-1706-0/krb5kdc/kudu_127.1.170.129.keytab
--principal=kudu/127.1.170.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.170.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790194121577027-1706-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.170.129
--webserver_port=0
--tserver_master_addrs=127.1.170.190:45463
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.170.129
--log_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790194121577027-1706-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:09:21.373054 5902 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:09:21.374841 5902 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:09:21.389436 5907 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:09:21.389394 5910 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:09:21.389484 5908 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:09:21.396195 5902 server_base.cc:1034] running on GCE node
I20260923 20:09:21.397282 5902 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:09:21.399354 5902 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:09:21.400643 5902 hybrid_clock.cc:648] HybridClock initialized: now 1790194161400582 us; error 57 us; skew 500 ppm
I20260923 20:09:21.401100 5902 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:09:21 dist-test-slave-th4g krb5kdc[5797](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194161, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.170.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:09:21.404568 5902 init.cc:377] Logged in from keytab as kudu/127.1.170.129@KRBTEST.COM (short username kudu)
I20260923 20:09:21.405979 5902 fs_manager.cc:359] Metadata directory not provided
I20260923 20:09:21.406077 5902 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:09:21.406354 5902 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:09:21.409461 5902 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790194121577027-1706-0/minicluster-data/ts-0/data/instance:
uuid: "50ae4ca35842404aa6740bb97b073195"
format_stamp: "Formatted at 2026-09-23 20:09:21 on dist-test-slave-th4g"
server_key: "d3096be53a8f692707cdb5ebfdf9344a"
server_key_iv: "364391d8748713e3ee4fe2a35c27454f"
server_key_version: "encryptionkey@0"
I20260923 20:09:21.410359 5902 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790194121577027-1706-0/minicluster-data/ts-0/wal/instance:
uuid: "50ae4ca35842404aa6740bb97b073195"
format_stamp: "Formatted at 2026-09-23 20:09:21 on dist-test-slave-th4g"
server_key: "d3096be53a8f692707cdb5ebfdf9344a"
server_key_iv: "364391d8748713e3ee4fe2a35c27454f"
server_key_version: "encryptionkey@0"
I20260923 20:09:21.415722 5902 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260923 20:09:21.419739 5916 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:09:21.421860 5902 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260923 20:09:21.422186 5902 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790194121577027-1706-0/minicluster-data/ts-0/data,/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790194121577027-1706-0/minicluster-data/ts-0/wal
uuid: "50ae4ca35842404aa6740bb97b073195"
format_stamp: "Formatted at 2026-09-23 20:09:21 on dist-test-slave-th4g"
server_key: "d3096be53a8f692707cdb5ebfdf9344a"
server_key_iv: "364391d8748713e3ee4fe2a35c27454f"
server_key_version: "encryptionkey@0"
I20260923 20:09:21.422672 5902 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790194121577027-1706-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790194121577027-1706-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790194121577027-1706-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:09:21.456852 5902 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:09:21.462208 5902 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:09:21.462528 5902 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:09:21.463652 5902 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 20:09:21.465108 5902 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 20:09:21.465240 5902 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 20:09:21.465370 5902 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 20:09:21.465459 5902 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 20:09:21.502202 5902 rpc_server.cc:307] RPC server started. Bound to: 127.1.170.129:34455
I20260923 20:09:21.502295 6028 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.170.129:34455 every 8 connection(s)
I20260923 20:09:21.503095 5902 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790194121577027-1706-0/minicluster-data/ts-0/data/info.pb
I20260923 20:09:21.511111 1706 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqUuku6/build/asan/bin/kudu as pid 5902
I20260923 20:09:21.511322 1706 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790194121577027-1706-0/minicluster-data/ts-0/wal/instance
I20260923 20:09:21.512189 1706 external_mini_cluster.cc:1428] Setting key f92341cf10a5430d2de79fc1d7d31e60
Sep 23 20:09:21 dist-test-slave-th4g krb5kdc[5797](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194161, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.170.129@KRBTEST.COM for kudu/127.1.170.190@KRBTEST.COM
I20260923 20:09:21.525195 6031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:09:21.505680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.170.129:42045 (local address 127.1.170.190:45463)
0923 20:09:21.506185 (+ 505us) server_negotiation.cc:206] Beginning negotiation
0923 20:09:21.506190 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 20:09:21.507017 (+ 827us) server_negotiation.cc:407] Connection header received
0923 20:09:21.508809 (+ 1792us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:09:21.508815 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:09:21.508903 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:09:21.509018 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:09:21.510578 (+ 1560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:09:21.511349 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:09:21.513651 (+ 2302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:09:21.514307 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:09:21.514463 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:09:21.518993 (+ 4530us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:09:21.519003 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:09:21.519010 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:09:21.519058 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:09:21.522094 (+ 3036us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:09:21.522891 (+ 797us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:09:21.522896 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:09:21.522898 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:09:21.523011 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:09:21.523434 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:09:21.523439 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:09:21.523442 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:09:21.523724 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:09:21.523877 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0923 20:09:21.524596 (+ 719us) server_negotiation.cc:299] Negotiation successful
0923 20:09:21.524831 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":395,"thread_start_us":274,"threads_started":1}
I20260923 20:09:21.526262 6032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:09:21.506153 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.190:45463 (local address 127.1.170.129:42045)
0923 20:09:21.506848 (+ 695us) negotiation.cc:107] Waiting for socket to connect
0923 20:09:21.506901 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0923 20:09:21.508540 (+ 1639us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:09:21.509255 (+ 715us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:09:21.509264 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:09:21.509779 (+ 515us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:09:21.510411 (+ 632us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:09:21.510427 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:09:21.511535 (+ 1108us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:09:21.511542 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:09:21.513496 (+ 1954us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:09:21.513509 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:09:21.514607 (+ 1098us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:09:21.514615 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:09:21.514919 (+ 304us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:09:21.515670 (+ 751us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:09:21.515708 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:09:21.518697 (+ 2989us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:09:21.522264 (+ 3567us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:09:21.522273 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:09:21.522293 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:09:21.522752 (+ 459us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:09:21.523118 (+ 366us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:09:21.523124 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:09:21.523128 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:09:21.523317 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:09:21.523849 (+ 532us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:09:21.523857 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:09:21.524210 (+ 353us) client_negotiation.cc:769] Sending connection context
0923 20:09:21.524490 (+ 280us) client_negotiation.cc:240] Negotiation successful
0923 20:09:21.525030 (+ 540us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":469,"thread_start_us":281,"threads_started":1}
I20260923 20:09:21.528028 6029 heartbeater.cc:347] Connected to a master server at 127.1.170.190:45463
I20260923 20:09:21.528345 6029 heartbeater.cc:464] Registering TS with master...
I20260923 20:09:21.529007 6029 heartbeater.cc:511] Master 127.1.170.190:45463 requested a full tablet report, sending...
I20260923 20:09:21.531383 5841 ts_manager.cc:194] Registered new tserver with Master: 50ae4ca35842404aa6740bb97b073195 (127.1.170.129:34455)
WARNING: no policy specified for kudu/127.1.170.130@KRBTEST.COM; defaulting to no policy
I20260923 20:09:21.533475 5841 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.170.129@KRBTEST.COM'} at 127.1.170.129:42045
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.170.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.170.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790194121577027-1706-0/krb5kdc/kudu_127.1.170.130.keytab.
Entry for principal kudu/127.1.170.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790194121577027-1706-0/krb5kdc/kudu_127.1.170.130.keytab.
I20260923 20:09:21.561767 1706 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqUuku6/build/asan/bin/kudu
/tmp/dist-test-taskqUuku6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790194121577027-1706-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790194121577027-1706-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790194121577027-1706-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790194121577027-1706-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.170.130:0
--local_ip_for_outbound_sockets=127.1.170.130
--webserver_interface=127.1.170.130
--webserver_port=0
--tserver_master_addrs=127.1.170.190:45463
--keytab_file=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790194121577027-1706-0/krb5kdc/kudu_127.1.170.130.keytab
--principal=kudu/127.1.170.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.170.148:45765
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790194121577027-1706-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790194121577027-1706-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790194121577027-1706-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:09:21.914556 6035 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:09:21.914855 6035 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:09:21.915023 6035 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:09:21.922809 6035 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:09:21.922920 6035 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:09:21.923084 6035 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.170.130
I20260923 20:09:21.933425 6035 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.170.148:45765
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790194121577027-1706-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790194121577027-1706-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790194121577027-1706-0/krb5kdc/kudu_127.1.170.130.keytab
--principal=kudu/127.1.170.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.170.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790194121577027-1706-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.170.130
--webserver_port=0
--tserver_master_addrs=127.1.170.190:45463
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.170.130
--log_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790194121577027-1706-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:09:21.934849 6035 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:09:21.936169 6035 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:09:21.950210 6043 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:09:21.950277 6041 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:09:21.950726 6040 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:09:21.955238 6035 server_base.cc:1034] running on GCE node
I20260923 20:09:21.955906 6035 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:09:21.956923 6035 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:09:21.958171 6035 hybrid_clock.cc:648] HybridClock initialized: now 1790194161958126 us; error 47 us; skew 500 ppm
I20260923 20:09:21.958508 6035 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:09:21 dist-test-slave-th4g krb5kdc[5797](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194161, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.170.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:09:21.962957 6035 init.cc:377] Logged in from keytab as kudu/127.1.170.130@KRBTEST.COM (short username kudu)
I20260923 20:09:21.964231 6035 fs_manager.cc:359] Metadata directory not provided
I20260923 20:09:21.964322 6035 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:09:21.964572 6035 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:09:21.967505 6035 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790194121577027-1706-0/minicluster-data/ts-1/data/instance:
uuid: "2637c8f627454dd78eff0b791765a030"
format_stamp: "Formatted at 2026-09-23 20:09:21 on dist-test-slave-th4g"
server_key: "786169bc7b78415a75cca9f3f8fe5751"
server_key_iv: "516d334aabcde0fe2f69b0e8f34edffd"
server_key_version: "encryptionkey@0"
I20260923 20:09:21.968369 6035 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790194121577027-1706-0/minicluster-data/ts-1/wal/instance:
uuid: "2637c8f627454dd78eff0b791765a030"
format_stamp: "Formatted at 2026-09-23 20:09:21 on dist-test-slave-th4g"
server_key: "786169bc7b78415a75cca9f3f8fe5751"
server_key_iv: "516d334aabcde0fe2f69b0e8f34edffd"
server_key_version: "encryptionkey@0"
I20260923 20:09:21.973951 6035 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260923 20:09:21.977558 6049 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:09:21.979405 6035 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260923 20:09:21.979560 6035 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790194121577027-1706-0/minicluster-data/ts-1/data,/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790194121577027-1706-0/minicluster-data/ts-1/wal
uuid: "2637c8f627454dd78eff0b791765a030"
format_stamp: "Formatted at 2026-09-23 20:09:21 on dist-test-slave-th4g"
server_key: "786169bc7b78415a75cca9f3f8fe5751"
server_key_iv: "516d334aabcde0fe2f69b0e8f34edffd"
server_key_version: "encryptionkey@0"
I20260923 20:09:21.979765 6035 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790194121577027-1706-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790194121577027-1706-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790194121577027-1706-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:09:21.995112 6035 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:09:21.999783 6035 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:09:22.000022 6035 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:09:22.001017 6035 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 20:09:22.002401 6035 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 20:09:22.002489 6035 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 20:09:22.002571 6035 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 20:09:22.002625 6035 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 20:09:22.037020 6035 rpc_server.cc:307] RPC server started. Bound to: 127.1.170.130:45517
I20260923 20:09:22.037122 6161 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.170.130:45517 every 8 connection(s)
I20260923 20:09:22.037928 6035 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790194121577027-1706-0/minicluster-data/ts-1/data/info.pb
I20260923 20:09:22.041539 1706 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqUuku6/build/asan/bin/kudu as pid 6035
I20260923 20:09:22.041713 1706 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790194121577027-1706-0/minicluster-data/ts-1/wal/instance
I20260923 20:09:22.042186 1706 external_mini_cluster.cc:1428] Setting key 524b439651526b705fe683d9d2d47d7b
Sep 23 20:09:22 dist-test-slave-th4g krb5kdc[5797](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194161, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.170.130@KRBTEST.COM for kudu/127.1.170.190@KRBTEST.COM
I20260923 20:09:22.057766 6164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:09:22.040274 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.170.130:44329 (local address 127.1.170.190:45463)
0923 20:09:22.040735 (+ 461us) server_negotiation.cc:206] Beginning negotiation
0923 20:09:22.040742 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:09:22.041630 (+ 888us) server_negotiation.cc:407] Connection header received
0923 20:09:22.043558 (+ 1928us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:09:22.043563 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:09:22.043647 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:09:22.043765 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:09:22.045206 (+ 1441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:09:22.046002 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:09:22.047533 (+ 1531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:09:22.047967 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:09:22.048090 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:09:22.051783 (+ 3693us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:09:22.051794 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:09:22.051799 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:09:22.051833 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:09:22.054781 (+ 2948us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:09:22.055527 (+ 746us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:09:22.055534 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:09:22.055537 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:09:22.055658 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:09:22.056060 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:09:22.056066 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:09:22.056070 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:09:22.056360 (+ 290us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:09:22.056505 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0923 20:09:22.057186 (+ 681us) server_negotiation.cc:299] Negotiation successful
0923 20:09:22.057399 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":333,"thread_start_us":219,"threads_started":1}
I20260923 20:09:22.058794 6165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:09:22.040717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.190:45463 (local address 127.1.170.130:44329)
0923 20:09:22.041461 (+ 744us) negotiation.cc:107] Waiting for socket to connect
0923 20:09:22.041496 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0923 20:09:22.043294 (+ 1798us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:09:22.043991 (+ 697us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:09:22.043999 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:09:22.044502 (+ 503us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:09:22.045045 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:09:22.045061 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:09:22.046146 (+ 1085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:09:22.046150 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:09:22.047385 (+ 1235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:09:22.047396 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:09:22.048200 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:09:22.048207 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:09:22.048413 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:09:22.049077 (+ 664us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:09:22.049103 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:09:22.051547 (+ 2444us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:09:22.054938 (+ 3391us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:09:22.054947 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:09:22.054966 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:09:22.055404 (+ 438us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:09:22.055766 (+ 362us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:09:22.055771 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:09:22.055775 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:09:22.055949 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:09:22.056492 (+ 543us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:09:22.056499 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:09:22.056837 (+ 338us) client_negotiation.cc:769] Sending connection context
0923 20:09:22.057103 (+ 266us) client_negotiation.cc:240] Negotiation successful
0923 20:09:22.057635 (+ 532us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":521,"thread_start_us":297,"threads_started":1}
WARNING: no policy specified for kudu/127.1.170.131@KRBTEST.COM; defaulting to no policy
I20260923 20:09:22.060374 6162 heartbeater.cc:347] Connected to a master server at 127.1.170.190:45463
I20260923 20:09:22.060702 6162 heartbeater.cc:464] Registering TS with master...
I20260923 20:09:22.061481 6162 heartbeater.cc:511] Master 127.1.170.190:45463 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.170.131@KRBTEST.COM" created.
I20260923 20:09:22.063113 5841 ts_manager.cc:194] Registered new tserver with Master: 2637c8f627454dd78eff0b791765a030 (127.1.170.130:45517)
I20260923 20:09:22.063938 5841 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.170.130@KRBTEST.COM'} at 127.1.170.130:44329
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.170.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790194121577027-1706-0/krb5kdc/kudu_127.1.170.131.keytab.
Entry for principal kudu/127.1.170.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790194121577027-1706-0/krb5kdc/kudu_127.1.170.131.keytab.
I20260923 20:09:22.090107 1706 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqUuku6/build/asan/bin/kudu
/tmp/dist-test-taskqUuku6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790194121577027-1706-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790194121577027-1706-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790194121577027-1706-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790194121577027-1706-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.170.131:0
--local_ip_for_outbound_sockets=127.1.170.131
--webserver_interface=127.1.170.131
--webserver_port=0
--tserver_master_addrs=127.1.170.190:45463
--keytab_file=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790194121577027-1706-0/krb5kdc/kudu_127.1.170.131.keytab
--principal=kudu/127.1.170.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.170.148:45765
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790194121577027-1706-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790194121577027-1706-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790194121577027-1706-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:09:22.414944 6168 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:09:22.415238 6168 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:09:22.415401 6168 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:09:22.422971 6168 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:09:22.423082 6168 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:09:22.423290 6168 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.170.131
I20260923 20:09:22.433499 6168 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.170.148:45765
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790194121577027-1706-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790194121577027-1706-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790194121577027-1706-0/krb5kdc/kudu_127.1.170.131.keytab
--principal=kudu/127.1.170.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.170.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790194121577027-1706-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.170.131
--webserver_port=0
--tserver_master_addrs=127.1.170.190:45463
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.170.131
--log_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790194121577027-1706-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:09:22.434975 6168 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:09:22.436219 6168 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:09:22.449321 6173 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:09:22.449242 6176 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:09:22.449268 6174 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:09:22.450088 6168 server_base.cc:1034] running on GCE node
I20260923 20:09:22.450651 6168 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:09:22.451552 6168 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:09:22.452750 6168 hybrid_clock.cc:648] HybridClock initialized: now 1790194162452699 us; error 39 us; skew 500 ppm
I20260923 20:09:22.453073 6168 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:09:22 dist-test-slave-th4g krb5kdc[5797](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194162, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.170.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:09:22.456168 6168 init.cc:377] Logged in from keytab as kudu/127.1.170.131@KRBTEST.COM (short username kudu)
I20260923 20:09:22.457310 6168 fs_manager.cc:359] Metadata directory not provided
I20260923 20:09:22.457394 6168 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:09:22.457638 6168 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:09:22.460592 6168 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790194121577027-1706-0/minicluster-data/ts-2/data/instance:
uuid: "9dad766957e545bf89d81dc1f7d9bb12"
format_stamp: "Formatted at 2026-09-23 20:09:22 on dist-test-slave-th4g"
server_key: "531308bbcd6c8cd6c74ad161f3783d3d"
server_key_iv: "748101885fc0d3d49e334d586f582a2e"
server_key_version: "encryptionkey@0"
I20260923 20:09:22.461400 6168 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790194121577027-1706-0/minicluster-data/ts-2/wal/instance:
uuid: "9dad766957e545bf89d81dc1f7d9bb12"
format_stamp: "Formatted at 2026-09-23 20:09:22 on dist-test-slave-th4g"
server_key: "531308bbcd6c8cd6c74ad161f3783d3d"
server_key_iv: "748101885fc0d3d49e334d586f582a2e"
server_key_version: "encryptionkey@0"
I20260923 20:09:22.466512 6168 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260923 20:09:22.470114 6182 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:09:22.471529 6168 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260923 20:09:22.471715 6168 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790194121577027-1706-0/minicluster-data/ts-2/data,/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790194121577027-1706-0/minicluster-data/ts-2/wal
uuid: "9dad766957e545bf89d81dc1f7d9bb12"
format_stamp: "Formatted at 2026-09-23 20:09:22 on dist-test-slave-th4g"
server_key: "531308bbcd6c8cd6c74ad161f3783d3d"
server_key_iv: "748101885fc0d3d49e334d586f582a2e"
server_key_version: "encryptionkey@0"
I20260923 20:09:22.471974 6168 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790194121577027-1706-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790194121577027-1706-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790194121577027-1706-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:09:22.500765 6168 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:09:22.505138 6168 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:09:22.505424 6168 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:09:22.506366 6168 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 20:09:22.507668 6168 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 20:09:22.507786 6168 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 20:09:22.507907 6168 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 20:09:22.507989 6168 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 20:09:22.537315 6029 heartbeater.cc:503] Master 127.1.170.190:45463 was elected leader, sending a full tablet report...
I20260923 20:09:22.541154 6168 rpc_server.cc:307] RPC server started. Bound to: 127.1.170.131:35633
I20260923 20:09:22.541249 6294 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.170.131:35633 every 8 connection(s)
I20260923 20:09:22.542013 6168 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790194121577027-1706-0/minicluster-data/ts-2/data/info.pb
I20260923 20:09:22.549737 1706 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqUuku6/build/asan/bin/kudu as pid 6168
I20260923 20:09:22.550056 1706 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790194121577027-1706-0/minicluster-data/ts-2/wal/instance
I20260923 20:09:22.550645 1706 external_mini_cluster.cc:1428] Setting key 79392291e746a6fced60fb4bd9521717
Sep 23 20:09:22 dist-test-slave-th4g krb5kdc[5797](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194162, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.170.131@KRBTEST.COM for kudu/127.1.170.190@KRBTEST.COM
I20260923 20:09:22.562177 6164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:09:22.544358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.170.131:49293 (local address 127.1.170.190:45463)
0923 20:09:22.544528 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0923 20:09:22.544533 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 20:09:22.545622 (+ 1089us) server_negotiation.cc:407] Connection header received
0923 20:09:22.547601 (+ 1979us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:09:22.547606 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:09:22.547699 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:09:22.547785 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:09:22.549290 (+ 1505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:09:22.550222 (+ 932us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:09:22.552429 (+ 2207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:09:22.552939 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:09:22.553036 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:09:22.556843 (+ 3807us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:09:22.556848 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:09:22.556851 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:09:22.556883 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:09:22.559261 (+ 2378us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:09:22.559953 (+ 692us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:09:22.559959 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:09:22.559962 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:09:22.560083 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:09:22.560504 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:09:22.560512 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:09:22.560515 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:09:22.560775 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:09:22.560932 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0923 20:09:22.561580 (+ 648us) server_negotiation.cc:299] Negotiation successful
0923 20:09:22.561805 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260923 20:09:22.563210 6297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:09:22.544799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.190:45463 (local address 127.1.170.131:49293)
0923 20:09:22.545460 (+ 661us) negotiation.cc:107] Waiting for socket to connect
0923 20:09:22.545498 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0923 20:09:22.547258 (+ 1760us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:09:22.548014 (+ 756us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:09:22.548023 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:09:22.548551 (+ 528us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:09:22.549108 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:09:22.549125 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:09:22.550395 (+ 1270us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:09:22.550401 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:09:22.552283 (+ 1882us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:09:22.552299 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:09:22.553213 (+ 914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:09:22.553222 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:09:22.553534 (+ 312us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:09:22.554374 (+ 840us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:09:22.554397 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:09:22.556628 (+ 2231us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:09:22.559384 (+ 2756us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:09:22.559391 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:09:22.559404 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:09:22.559818 (+ 414us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:09:22.560190 (+ 372us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:09:22.560194 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:09:22.560198 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:09:22.560397 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:09:22.560898 (+ 501us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:09:22.560906 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:09:22.561253 (+ 347us) client_negotiation.cc:769] Sending connection context
0923 20:09:22.561523 (+ 270us) client_negotiation.cc:240] Negotiation successful
0923 20:09:22.562078 (+ 555us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":441,"thread_start_us":258,"threads_started":1}
I20260923 20:09:22.564674 6295 heartbeater.cc:347] Connected to a master server at 127.1.170.190:45463
I20260923 20:09:22.564972 6295 heartbeater.cc:464] Registering TS with master...
I20260923 20:09:22.565701 6295 heartbeater.cc:511] Master 127.1.170.190:45463 requested a full tablet report, sending...
I20260923 20:09:22.567076 5841 ts_manager.cc:194] Registered new tserver with Master: 9dad766957e545bf89d81dc1f7d9bb12 (127.1.170.131:35633)
I20260923 20:09:22.567854 5841 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.170.131@KRBTEST.COM'} at 127.1.170.131:49293
I20260923 20:09:22.578675 1706 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260923 20:09:22.594802 6164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:09:22.583011 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60916 (local address 127.1.170.190:45463)
0923 20:09:22.583204 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0923 20:09:22.583208 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 20:09:22.583723 (+ 515us) server_negotiation.cc:407] Connection header received
0923 20:09:22.583892 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:09:22.583896 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:09:22.583957 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:09:22.584034 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:09:22.585394 (+ 1360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:09:22.586153 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:09:22.587391 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:09:22.587808 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:09:22.587900 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:09:22.590075 (+ 2175us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:09:22.590080 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:09:22.590084 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:09:22.590112 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:09:22.592543 (+ 2431us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:09:22.593161 (+ 618us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:09:22.593166 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:09:22.593168 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:09:22.593310 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:09:22.593606 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:09:22.593609 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:09:22.593611 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:09:22.593959 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:09:22.594165 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0923 20:09:22.594401 (+ 236us) server_negotiation.cc:299] Negotiation successful
0923 20:09:22.594553 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260923 20:09:22.598774 5841 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60916:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 20:09:22.605381 5841 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 20:09:22.628032 6310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:09:22.617808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39830 (local address 127.1.170.130:45517)
0923 20:09:22.618955 (+ 1147us) server_negotiation.cc:206] Beginning negotiation
0923 20:09:22.618968 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0923 20:09:22.619508 (+ 540us) server_negotiation.cc:407] Connection header received
0923 20:09:22.619861 (+ 353us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:09:22.619873 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:09:22.620126 (+ 253us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:09:22.620283 (+ 157us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:09:22.621082 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:09:22.622569 (+ 1487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:09:22.625096 (+ 2527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:09:22.626590 (+ 1494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:09:22.626721 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:09:22.626925 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0923 20:09:22.627510 (+ 585us) server_negotiation.cc:299] Negotiation successful
0923 20:09:22.627678 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":944,"thread_start_us":1536,"threads_started":1}
I20260923 20:09:22.628338 6308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:09:22.616793 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38158 (local address 127.1.170.129:34455)
0923 20:09:22.620786 (+ 3993us) server_negotiation.cc:206] Beginning negotiation
0923 20:09:22.620794 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 20:09:22.620821 (+ 27us) server_negotiation.cc:407] Connection header received
0923 20:09:22.620912 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:09:22.620921 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:09:22.621132 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:09:22.621320 (+ 188us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:09:22.621928 (+ 608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:09:22.623324 (+ 1396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:09:22.625828 (+ 2504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:09:22.627285 (+ 1457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:09:22.627437 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:09:22.627701 (+ 264us) server_negotiation.cc:1035] Waiting for connection context
0923 20:09:22.627834 (+ 133us) server_negotiation.cc:299] Negotiation successful
0923 20:09:22.628002 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3834,"thread_start_us":3679,"threads_started":1}
I20260923 20:09:22.628512 6309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:09:22.616585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.129:34455 (local address 127.0.0.1:38158)
0923 20:09:22.618010 (+ 1425us) negotiation.cc:107] Waiting for socket to connect
0923 20:09:22.618070 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0923 20:09:22.618272 (+ 202us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:09:22.621310 (+ 3038us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:09:22.621314 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:09:22.621338 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:09:22.621698 (+ 360us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:09:22.621710 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:09:22.623470 (+ 1760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:09:22.623476 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:09:22.625674 (+ 2198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:09:22.625689 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:09:22.627476 (+ 1787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:09:22.627482 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:09:22.627647 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:09:22.627676 (+ 29us) client_negotiation.cc:769] Sending connection context
0923 20:09:22.627762 (+ 86us) client_negotiation.cc:240] Negotiation successful
0923 20:09:22.627956 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1077,"mutex_wait_us":107,"thread_start_us":2566,"threads_started":1}
I20260923 20:09:22.628875 6307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:09:22.617201 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.130:45517 (local address 127.0.0.1:39830)
0923 20:09:22.619411 (+ 2210us) negotiation.cc:107] Waiting for socket to connect
0923 20:09:22.619433 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0923 20:09:22.619608 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:09:22.620589 (+ 981us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:09:22.620596 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:09:22.620625 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:09:22.620952 (+ 327us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:09:22.620963 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:09:22.622770 (+ 1807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:09:22.622775 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:09:22.624941 (+ 2166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:09:22.624955 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:09:22.627045 (+ 2090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:09:22.627050 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:09:22.627239 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:09:22.627292 (+ 53us) client_negotiation.cc:769] Sending connection context
0923 20:09:22.628293 (+ 1001us) client_negotiation.cc:240] Negotiation successful
0923 20:09:22.628438 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2089,"mutex_wait_us":3,"thread_start_us":815,"threads_started":1}
I20260923 20:09:22.629318 6311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:09:22.619133 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.131:35633 (local address 127.0.0.1:60950)
0923 20:09:22.620146 (+ 1013us) negotiation.cc:107] Waiting for socket to connect
0923 20:09:22.620164 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0923 20:09:22.620277 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:09:22.621657 (+ 1380us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:09:22.621665 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:09:22.621693 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:09:22.622032 (+ 339us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:09:22.622044 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:09:22.623930 (+ 1886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:09:22.623935 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:09:22.625953 (+ 2018us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:09:22.625967 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:09:22.627970 (+ 2003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:09:22.627976 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:09:22.628806 (+ 830us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:09:22.628841 (+ 35us) client_negotiation.cc:769] Sending connection context
0923 20:09:22.628952 (+ 111us) client_negotiation.cc:240] Negotiation successful
0923 20:09:22.629090 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":897,"thread_start_us":973,"threads_started":1}
I20260923 20:09:22.630106 6312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:09:22.619837 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60950 (local address 127.1.170.131:35633)
0923 20:09:22.621133 (+ 1296us) server_negotiation.cc:206] Beginning negotiation
0923 20:09:22.621143 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0923 20:09:22.621176 (+ 33us) server_negotiation.cc:407] Connection header received
0923 20:09:22.621269 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:09:22.621278 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:09:22.621470 (+ 192us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:09:22.621656 (+ 186us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:09:22.622228 (+ 572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:09:22.623795 (+ 1567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:09:22.626097 (+ 2302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:09:22.627566 (+ 1469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:09:22.627688 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:09:22.629594 (+ 1906us) server_negotiation.cc:1035] Waiting for connection context
0923 20:09:22.629653 (+ 59us) server_negotiation.cc:299] Negotiation successful
0923 20:09:22.629755 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1142,"thread_start_us":992,"threads_started":1}
I20260923 20:09:22.632285 5964 tablet_service.cc:1467] Processing CreateTablet for tablet 26f70eac938142a88e3f455efee5628e (DEFAULT_TABLE table=test-table [id=1b099ea8a0b94292a88ccee050c4bfca]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 20:09:22.632946 6097 tablet_service.cc:1467] Processing CreateTablet for tablet 26f70eac938142a88e3f455efee5628e (DEFAULT_TABLE table=test-table [id=1b099ea8a0b94292a88ccee050c4bfca]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 20:09:22.633514 5964 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 26f70eac938142a88e3f455efee5628e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:09:22.633292 6230 tablet_service.cc:1467] Processing CreateTablet for tablet 26f70eac938142a88e3f455efee5628e (DEFAULT_TABLE table=test-table [id=1b099ea8a0b94292a88ccee050c4bfca]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 20:09:22.634239 6097 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 26f70eac938142a88e3f455efee5628e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:09:22.634475 6230 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 26f70eac938142a88e3f455efee5628e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:09:22.644892 6313 tablet_bootstrap.cc:492] T 26f70eac938142a88e3f455efee5628e P 50ae4ca35842404aa6740bb97b073195: Bootstrap starting.
I20260923 20:09:22.648115 6313 tablet_bootstrap.cc:654] T 26f70eac938142a88e3f455efee5628e P 50ae4ca35842404aa6740bb97b073195: Neither blocks nor log segments found. Creating new log.
I20260923 20:09:22.648550 6314 tablet_bootstrap.cc:492] T 26f70eac938142a88e3f455efee5628e P 2637c8f627454dd78eff0b791765a030: Bootstrap starting.
I20260923 20:09:22.649245 6315 tablet_bootstrap.cc:492] T 26f70eac938142a88e3f455efee5628e P 9dad766957e545bf89d81dc1f7d9bb12: Bootstrap starting.
I20260923 20:09:22.651057 6313 log.cc:824] T 26f70eac938142a88e3f455efee5628e P 50ae4ca35842404aa6740bb97b073195: Log is configured to *not* fsync() on all Append() calls
I20260923 20:09:22.652460 6315 tablet_bootstrap.cc:654] T 26f70eac938142a88e3f455efee5628e P 9dad766957e545bf89d81dc1f7d9bb12: Neither blocks nor log segments found. Creating new log.
I20260923 20:09:22.652493 6314 tablet_bootstrap.cc:654] T 26f70eac938142a88e3f455efee5628e P 2637c8f627454dd78eff0b791765a030: Neither blocks nor log segments found. Creating new log.
I20260923 20:09:22.653533 6314 log.cc:824] T 26f70eac938142a88e3f455efee5628e P 2637c8f627454dd78eff0b791765a030: Log is configured to *not* fsync() on all Append() calls
I20260923 20:09:22.653673 6315 log.cc:824] T 26f70eac938142a88e3f455efee5628e P 9dad766957e545bf89d81dc1f7d9bb12: Log is configured to *not* fsync() on all Append() calls
I20260923 20:09:22.654482 6313 tablet_bootstrap.cc:492] T 26f70eac938142a88e3f455efee5628e P 50ae4ca35842404aa6740bb97b073195: No bootstrap required, opened a new log
I20260923 20:09:22.654764 6313 ts_tablet_manager.cc:1397] T 26f70eac938142a88e3f455efee5628e P 50ae4ca35842404aa6740bb97b073195: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260923 20:09:22.656243 6315 tablet_bootstrap.cc:492] T 26f70eac938142a88e3f455efee5628e P 9dad766957e545bf89d81dc1f7d9bb12: No bootstrap required, opened a new log
I20260923 20:09:22.656454 6314 tablet_bootstrap.cc:492] T 26f70eac938142a88e3f455efee5628e P 2637c8f627454dd78eff0b791765a030: No bootstrap required, opened a new log
I20260923 20:09:22.656524 6315 ts_tablet_manager.cc:1397] T 26f70eac938142a88e3f455efee5628e P 9dad766957e545bf89d81dc1f7d9bb12: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260923 20:09:22.656688 6314 ts_tablet_manager.cc:1397] T 26f70eac938142a88e3f455efee5628e P 2637c8f627454dd78eff0b791765a030: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.003s
I20260923 20:09:22.663949 6315 raft_consensus.cc:348] T 26f70eac938142a88e3f455efee5628e P 9dad766957e545bf89d81dc1f7d9bb12 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50ae4ca35842404aa6740bb97b073195" member_type: VOTER last_known_addr { host: "127.1.170.129" port: 34455 } } peers { permanent_uuid: "2637c8f627454dd78eff0b791765a030" member_type: VOTER last_known_addr { host: "127.1.170.130" port: 45517 } } peers { permanent_uuid: "9dad766957e545bf89d81dc1f7d9bb12" member_type: VOTER last_known_addr { host: "127.1.170.131" port: 35633 } }
I20260923 20:09:22.664335 6315 raft_consensus.cc:374] T 26f70eac938142a88e3f455efee5628e P 9dad766957e545bf89d81dc1f7d9bb12 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:09:22.664484 6315 raft_consensus.cc:729] T 26f70eac938142a88e3f455efee5628e P 9dad766957e545bf89d81dc1f7d9bb12 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9dad766957e545bf89d81dc1f7d9bb12, State: Initialized, Role: FOLLOWER
I20260923 20:09:22.665097 6315 consensus_queue.cc:260] T 26f70eac938142a88e3f455efee5628e P 9dad766957e545bf89d81dc1f7d9bb12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50ae4ca35842404aa6740bb97b073195" member_type: VOTER last_known_addr { host: "127.1.170.129" port: 34455 } } peers { permanent_uuid: "2637c8f627454dd78eff0b791765a030" member_type: VOTER last_known_addr { host: "127.1.170.130" port: 45517 } } peers { permanent_uuid: "9dad766957e545bf89d81dc1f7d9bb12" member_type: VOTER last_known_addr { host: "127.1.170.131" port: 35633 } }
I20260923 20:09:22.665720 6313 raft_consensus.cc:348] T 26f70eac938142a88e3f455efee5628e P 50ae4ca35842404aa6740bb97b073195 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50ae4ca35842404aa6740bb97b073195" member_type: VOTER last_known_addr { host: "127.1.170.129" port: 34455 } } peers { permanent_uuid: "2637c8f627454dd78eff0b791765a030" member_type: VOTER last_known_addr { host: "127.1.170.130" port: 45517 } } peers { permanent_uuid: "9dad766957e545bf89d81dc1f7d9bb12" member_type: VOTER last_known_addr { host: "127.1.170.131" port: 35633 } }
I20260923 20:09:22.666211 6313 raft_consensus.cc:374] T 26f70eac938142a88e3f455efee5628e P 50ae4ca35842404aa6740bb97b073195 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:09:22.666244 6295 heartbeater.cc:503] Master 127.1.170.190:45463 was elected leader, sending a full tablet report...
I20260923 20:09:22.666374 6313 raft_consensus.cc:729] T 26f70eac938142a88e3f455efee5628e P 50ae4ca35842404aa6740bb97b073195 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 50ae4ca35842404aa6740bb97b073195, State: Initialized, Role: FOLLOWER
I20260923 20:09:22.667088 6315 ts_tablet_manager.cc:1428] T 26f70eac938142a88e3f455efee5628e P 9dad766957e545bf89d81dc1f7d9bb12: Time spent starting tablet: real 0.010s user 0.009s sys 0.001s
I20260923 20:09:22.666983 6313 consensus_queue.cc:260] T 26f70eac938142a88e3f455efee5628e P 50ae4ca35842404aa6740bb97b073195 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50ae4ca35842404aa6740bb97b073195" member_type: VOTER last_known_addr { host: "127.1.170.129" port: 34455 } } peers { permanent_uuid: "2637c8f627454dd78eff0b791765a030" member_type: VOTER last_known_addr { host: "127.1.170.130" port: 45517 } } peers { permanent_uuid: "9dad766957e545bf89d81dc1f7d9bb12" member_type: VOTER last_known_addr { host: "127.1.170.131" port: 35633 } }
I20260923 20:09:22.666878 6314 raft_consensus.cc:348] T 26f70eac938142a88e3f455efee5628e P 2637c8f627454dd78eff0b791765a030 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50ae4ca35842404aa6740bb97b073195" member_type: VOTER last_known_addr { host: "127.1.170.129" port: 34455 } } peers { permanent_uuid: "2637c8f627454dd78eff0b791765a030" member_type: VOTER last_known_addr { host: "127.1.170.130" port: 45517 } } peers { permanent_uuid: "9dad766957e545bf89d81dc1f7d9bb12" member_type: VOTER last_known_addr { host: "127.1.170.131" port: 35633 } }
I20260923 20:09:22.667271 6314 raft_consensus.cc:374] T 26f70eac938142a88e3f455efee5628e P 2637c8f627454dd78eff0b791765a030 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:09:22.667399 6314 raft_consensus.cc:729] T 26f70eac938142a88e3f455efee5628e P 2637c8f627454dd78eff0b791765a030 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2637c8f627454dd78eff0b791765a030, State: Initialized, Role: FOLLOWER
I20260923 20:09:22.668017 6314 consensus_queue.cc:260] T 26f70eac938142a88e3f455efee5628e P 2637c8f627454dd78eff0b791765a030 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50ae4ca35842404aa6740bb97b073195" member_type: VOTER last_known_addr { host: "127.1.170.129" port: 34455 } } peers { permanent_uuid: "2637c8f627454dd78eff0b791765a030" member_type: VOTER last_known_addr { host: "127.1.170.130" port: 45517 } } peers { permanent_uuid: "9dad766957e545bf89d81dc1f7d9bb12" member_type: VOTER last_known_addr { host: "127.1.170.131" port: 35633 } }
I20260923 20:09:22.669252 6162 heartbeater.cc:503] Master 127.1.170.190:45463 was elected leader, sending a full tablet report...
I20260923 20:09:22.670439 6314 ts_tablet_manager.cc:1428] T 26f70eac938142a88e3f455efee5628e P 2637c8f627454dd78eff0b791765a030: Time spent starting tablet: real 0.014s user 0.008s sys 0.007s
I20260923 20:09:22.671999 6313 ts_tablet_manager.cc:1428] T 26f70eac938142a88e3f455efee5628e P 50ae4ca35842404aa6740bb97b073195: Time spent starting tablet: real 0.017s user 0.013s sys 0.000s
I20260923 20:09:22.754652 6320 raft_consensus.cc:482] T 26f70eac938142a88e3f455efee5628e P 50ae4ca35842404aa6740bb97b073195 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 20:09:22.755034 6320 raft_consensus.cc:504] T 26f70eac938142a88e3f455efee5628e P 50ae4ca35842404aa6740bb97b073195 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50ae4ca35842404aa6740bb97b073195" member_type: VOTER last_known_addr { host: "127.1.170.129" port: 34455 } } peers { permanent_uuid: "2637c8f627454dd78eff0b791765a030" member_type: VOTER last_known_addr { host: "127.1.170.130" port: 45517 } } peers { permanent_uuid: "9dad766957e545bf89d81dc1f7d9bb12" member_type: VOTER last_known_addr { host: "127.1.170.131" port: 35633 } }
W20260923 20:09:22.755294 6030 tablet.cc:2369] T 26f70eac938142a88e3f455efee5628e P 50ae4ca35842404aa6740bb97b073195: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 20:09:22.756757 6320 leader_election.cc:288] T 26f70eac938142a88e3f455efee5628e P 50ae4ca35842404aa6740bb97b073195 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2637c8f627454dd78eff0b791765a030 (127.1.170.130:45517), 9dad766957e545bf89d81dc1f7d9bb12 (127.1.170.131:35633)
I20260923 20:09:22.763204 6310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:09:22.757116 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.170.129:50641 (local address 127.1.170.130:45517)
0923 20:09:22.757295 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0923 20:09:22.757299 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 20:09:22.757895 (+ 596us) server_negotiation.cc:407] Connection header received
0923 20:09:22.758142 (+ 247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:09:22.758147 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:09:22.758250 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:09:22.758366 (+ 116us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:09:22.758948 (+ 582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:09:22.759705 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:09:22.761420 (+ 1715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:09:22.762308 (+ 888us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:09:22.762467 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:09:22.762543 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0923 20:09:22.762824 (+ 281us) server_negotiation.cc:299] Negotiation successful
0923 20:09:22.762944 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260923 20:09:22.763360 6322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:09:22.757045 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.130:45517 (local address 127.1.170.129:50641)
0923 20:09:22.757743 (+ 698us) negotiation.cc:107] Waiting for socket to connect
0923 20:09:22.757769 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0923 20:09:22.757929 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:09:22.758426 (+ 497us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:09:22.758431 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:09:22.758463 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:09:22.758807 (+ 344us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:09:22.758816 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:09:22.759834 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:09:22.759842 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:09:22.761264 (+ 1422us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:09:22.761275 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:09:22.762500 (+ 1225us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:09:22.762506 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:09:22.762654 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:09:22.762677 (+ 23us) client_negotiation.cc:769] Sending connection context
0923 20:09:22.762801 (+ 124us) client_negotiation.cc:240] Negotiation successful
0923 20:09:22.762935 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":557,"thread_start_us":377,"threads_started":1}
I20260923 20:09:22.763646 6312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:09:22.758124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.170.129:57447 (local address 127.1.170.131:35633)
0923 20:09:22.758331 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0923 20:09:22.758339 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 20:09:22.758618 (+ 279us) server_negotiation.cc:407] Connection header received
0923 20:09:22.758816 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:09:22.758822 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:09:22.758883 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:09:22.758968 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:09:22.759493 (+ 525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:09:22.760171 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:09:22.761800 (+ 1629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:09:22.762690 (+ 890us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:09:22.762865 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:09:22.762941 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0923 20:09:22.763263 (+ 322us) server_negotiation.cc:299] Negotiation successful
0923 20:09:22.763422 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260923 20:09:22.764267 6323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:09:22.757928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.131:35633 (local address 127.1.170.129:57447)
0923 20:09:22.758514 (+ 586us) negotiation.cc:107] Waiting for socket to connect
0923 20:09:22.758530 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0923 20:09:22.758672 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:09:22.758990 (+ 318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:09:22.758994 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:09:22.759017 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:09:22.759354 (+ 337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:09:22.759361 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:09:22.760312 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:09:22.760317 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:09:22.761657 (+ 1340us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:09:22.761668 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:09:22.762914 (+ 1246us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:09:22.762921 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:09:22.763071 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:09:22.763093 (+ 22us) client_negotiation.cc:769] Sending connection context
0923 20:09:22.763830 (+ 737us) client_negotiation.cc:240] Negotiation successful
0923 20:09:22.763927 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":485,"thread_start_us":328,"threads_started":1}
I20260923 20:09:22.764487 6117 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26f70eac938142a88e3f455efee5628e" candidate_uuid: "50ae4ca35842404aa6740bb97b073195" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2637c8f627454dd78eff0b791765a030" is_pre_election: true
I20260923 20:09:22.765024 6117 raft_consensus.cc:2454] T 26f70eac938142a88e3f455efee5628e P 2637c8f627454dd78eff0b791765a030 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 50ae4ca35842404aa6740bb97b073195 in term 0.
I20260923 20:09:22.765236 6250 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26f70eac938142a88e3f455efee5628e" candidate_uuid: "50ae4ca35842404aa6740bb97b073195" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9dad766957e545bf89d81dc1f7d9bb12" is_pre_election: true
I20260923 20:09:22.765733 6250 raft_consensus.cc:2454] T 26f70eac938142a88e3f455efee5628e P 9dad766957e545bf89d81dc1f7d9bb12 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 50ae4ca35842404aa6740bb97b073195 in term 0.
I20260923 20:09:22.765749 5919 leader_election.cc:302] T 26f70eac938142a88e3f455efee5628e P 50ae4ca35842404aa6740bb97b073195 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2637c8f627454dd78eff0b791765a030, 50ae4ca35842404aa6740bb97b073195; no voters:
I20260923 20:09:22.766199 6320 raft_consensus.cc:2781] T 26f70eac938142a88e3f455efee5628e P 50ae4ca35842404aa6740bb97b073195 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 20:09:22.766420 6320 raft_consensus.cc:482] T 26f70eac938142a88e3f455efee5628e P 50ae4ca35842404aa6740bb97b073195 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 20:09:22.766532 6320 raft_consensus.cc:3037] T 26f70eac938142a88e3f455efee5628e P 50ae4ca35842404aa6740bb97b073195 [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:09:22.768661 6320 raft_consensus.cc:504] T 26f70eac938142a88e3f455efee5628e P 50ae4ca35842404aa6740bb97b073195 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50ae4ca35842404aa6740bb97b073195" member_type: VOTER last_known_addr { host: "127.1.170.129" port: 34455 } } peers { permanent_uuid: "2637c8f627454dd78eff0b791765a030" member_type: VOTER last_known_addr { host: "127.1.170.130" port: 45517 } } peers { permanent_uuid: "9dad766957e545bf89d81dc1f7d9bb12" member_type: VOTER last_known_addr { host: "127.1.170.131" port: 35633 } }
I20260923 20:09:22.769301 6320 leader_election.cc:288] T 26f70eac938142a88e3f455efee5628e P 50ae4ca35842404aa6740bb97b073195 [CANDIDATE]: Term 1 election: Requested vote from peers 2637c8f627454dd78eff0b791765a030 (127.1.170.130:45517), 9dad766957e545bf89d81dc1f7d9bb12 (127.1.170.131:35633)
I20260923 20:09:22.769685 6117 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26f70eac938142a88e3f455efee5628e" candidate_uuid: "50ae4ca35842404aa6740bb97b073195" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2637c8f627454dd78eff0b791765a030"
I20260923 20:09:22.769869 6117 raft_consensus.cc:3037] T 26f70eac938142a88e3f455efee5628e P 2637c8f627454dd78eff0b791765a030 [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:09:22.769914 6250 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26f70eac938142a88e3f455efee5628e" candidate_uuid: "50ae4ca35842404aa6740bb97b073195" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9dad766957e545bf89d81dc1f7d9bb12"
I20260923 20:09:22.770114 6250 raft_consensus.cc:3037] T 26f70eac938142a88e3f455efee5628e P 9dad766957e545bf89d81dc1f7d9bb12 [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:09:22.772425 6117 raft_consensus.cc:2454] T 26f70eac938142a88e3f455efee5628e P 2637c8f627454dd78eff0b791765a030 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 50ae4ca35842404aa6740bb97b073195 in term 1.
I20260923 20:09:22.772681 6250 raft_consensus.cc:2454] T 26f70eac938142a88e3f455efee5628e P 9dad766957e545bf89d81dc1f7d9bb12 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 50ae4ca35842404aa6740bb97b073195 in term 1.
I20260923 20:09:22.772806 5919 leader_election.cc:302] T 26f70eac938142a88e3f455efee5628e P 50ae4ca35842404aa6740bb97b073195 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2637c8f627454dd78eff0b791765a030, 50ae4ca35842404aa6740bb97b073195; no voters:
I20260923 20:09:22.773089 6320 raft_consensus.cc:2781] T 26f70eac938142a88e3f455efee5628e P 50ae4ca35842404aa6740bb97b073195 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 20:09:22.773535 6320 raft_consensus.cc:686] T 26f70eac938142a88e3f455efee5628e P 50ae4ca35842404aa6740bb97b073195 [term 1 LEADER]: Becoming Leader. State: Replica: 50ae4ca35842404aa6740bb97b073195, State: Running, Role: LEADER
I20260923 20:09:22.774082 6320 consensus_queue.cc:237] T 26f70eac938142a88e3f455efee5628e P 50ae4ca35842404aa6740bb97b073195 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50ae4ca35842404aa6740bb97b073195" member_type: VOTER last_known_addr { host: "127.1.170.129" port: 34455 } } peers { permanent_uuid: "2637c8f627454dd78eff0b791765a030" member_type: VOTER last_known_addr { host: "127.1.170.130" port: 45517 } } peers { permanent_uuid: "9dad766957e545bf89d81dc1f7d9bb12" member_type: VOTER last_known_addr { host: "127.1.170.131" port: 35633 } }
I20260923 20:09:22.778374 5841 catalog_manager.cc:5495] T 26f70eac938142a88e3f455efee5628e P 50ae4ca35842404aa6740bb97b073195 reported cstate change: term changed from 0 to 1, leader changed from <none> to 50ae4ca35842404aa6740bb97b073195 (127.1.170.129). New cstate: current_term: 1 leader_uuid: "50ae4ca35842404aa6740bb97b073195" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50ae4ca35842404aa6740bb97b073195" member_type: VOTER last_known_addr { host: "127.1.170.129" port: 34455 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2637c8f627454dd78eff0b791765a030" member_type: VOTER last_known_addr { host: "127.1.170.130" port: 45517 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9dad766957e545bf89d81dc1f7d9bb12" member_type: VOTER last_known_addr { host: "127.1.170.131" port: 35633 } health_report { overall_health: UNKNOWN } } }
W20260923 20:09:22.789691 6163 tablet.cc:2369] T 26f70eac938142a88e3f455efee5628e P 2637c8f627454dd78eff0b791765a030: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260923 20:09:22.793442 6296 tablet.cc:2369] T 26f70eac938142a88e3f455efee5628e P 9dad766957e545bf89d81dc1f7d9bb12: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 20:09:22.821866 6308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:09:22.815819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38174 (local address 127.1.170.129:34455)
0923 20:09:22.816057 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0923 20:09:22.816064 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:09:22.816176 (+ 112us) server_negotiation.cc:407] Connection header received
0923 20:09:22.816452 (+ 276us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:09:22.816456 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:09:22.816542 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:09:22.816699 (+ 157us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 20:09:22.817057 (+ 358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:09:22.817811 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:09:22.819782 (+ 1971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:09:22.820193 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:09:22.820313 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:09:22.820817 (+ 504us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 20:09:22.821342 (+ 525us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 20:09:22.821447 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0923 20:09:22.821570 (+ 123us) server_negotiation.cc:299] Negotiation successful
0923 20:09:22.821630 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260923 20:09:22.830674 6117 raft_consensus.cc:1261] T 26f70eac938142a88e3f455efee5628e P 2637c8f627454dd78eff0b791765a030 [term 1 FOLLOWER]: Refusing update from remote peer 50ae4ca35842404aa6740bb97b073195: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 20:09:22.830785 6250 raft_consensus.cc:1261] T 26f70eac938142a88e3f455efee5628e P 9dad766957e545bf89d81dc1f7d9bb12 [term 1 FOLLOWER]: Refusing update from remote peer 50ae4ca35842404aa6740bb97b073195: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 20:09:22.831682 6324 consensus_queue.cc:1035] T 26f70eac938142a88e3f455efee5628e P 50ae4ca35842404aa6740bb97b073195 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9dad766957e545bf89d81dc1f7d9bb12" member_type: VOTER last_known_addr { host: "127.1.170.131" port: 35633 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 20:09:22.832111 6320 consensus_queue.cc:1035] T 26f70eac938142a88e3f455efee5628e P 50ae4ca35842404aa6740bb97b073195 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2637c8f627454dd78eff0b791765a030" member_type: VOTER last_known_addr { host: "127.1.170.130" port: 45517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 20:09:22.843938 6327 mvcc.cc:205] Tried to move back new op lower bound from 7332635290944843776 to 7332635290725994496. Current Snapshot: MvccSnapshot[applied={T|T < 7332635290944843776}]
I20260923 20:09:22.858199 5840 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60916:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 20:09:22.858536 5840 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60916:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 20:09:22.862223 5840 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ea263cafa53b48238eb244aa75235503: Sending DeleteTablet for 3 replicas of tablet 26f70eac938142a88e3f455efee5628e
I20260923 20:09:22.863651 5964 tablet_service.cc:1514] Processing DeleteTablet for tablet 26f70eac938142a88e3f455efee5628e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 20:09:22 UTC) from {username='kudu', principal='kudu/127.1.170.190@KRBTEST.COM'} at 127.0.0.1:38158
I20260923 20:09:22.863672 6230 tablet_service.cc:1514] Processing DeleteTablet for tablet 26f70eac938142a88e3f455efee5628e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 20:09:22 UTC) from {username='kudu', principal='kudu/127.1.170.190@KRBTEST.COM'} at 127.0.0.1:60950
I20260923 20:09:22.863806 6097 tablet_service.cc:1514] Processing DeleteTablet for tablet 26f70eac938142a88e3f455efee5628e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 20:09:22 UTC) from {username='kudu', principal='kudu/127.1.170.190@KRBTEST.COM'} at 127.0.0.1:39830
I20260923 20:09:22.866039 1706 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskqUuku6/build/asan/bin/kudu pid 5902
I20260923 20:09:22.875094 6337 tablet_replica.cc:330] T 26f70eac938142a88e3f455efee5628e P 50ae4ca35842404aa6740bb97b073195: stopping tablet replica
I20260923 20:09:22.875753 6337 raft_consensus.cc:2229] T 26f70eac938142a88e3f455efee5628e P 50ae4ca35842404aa6740bb97b073195 [term 1 LEADER]: Raft consensus shutting down.
I20260923 20:09:22.877032 6339 tablet_replica.cc:330] T 26f70eac938142a88e3f455efee5628e P 2637c8f627454dd78eff0b791765a030: stopping tablet replica
I20260923 20:09:22.877151 6337 raft_consensus.cc:2258] T 26f70eac938142a88e3f455efee5628e P 50ae4ca35842404aa6740bb97b073195 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 20:09:22.878014 6339 raft_consensus.cc:2229] T 26f70eac938142a88e3f455efee5628e P 2637c8f627454dd78eff0b791765a030 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 20:09:22.878559 6339 raft_consensus.cc:2258] T 26f70eac938142a88e3f455efee5628e P 2637c8f627454dd78eff0b791765a030 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 20:09:22.879529 6338 tablet_replica.cc:330] T 26f70eac938142a88e3f455efee5628e P 9dad766957e545bf89d81dc1f7d9bb12: stopping tablet replica
I20260923 20:09:22.880720 6337 ts_tablet_manager.cc:1905] T 26f70eac938142a88e3f455efee5628e P 50ae4ca35842404aa6740bb97b073195: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 20:09:22.880842 6339 ts_tablet_manager.cc:1905] T 26f70eac938142a88e3f455efee5628e P 2637c8f627454dd78eff0b791765a030: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 20:09:22.882277 6338 raft_consensus.cc:2229] T 26f70eac938142a88e3f455efee5628e P 9dad766957e545bf89d81dc1f7d9bb12 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 20:09:22.882867 6338 raft_consensus.cc:2258] T 26f70eac938142a88e3f455efee5628e P 9dad766957e545bf89d81dc1f7d9bb12 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 20:09:22.886356 6338 ts_tablet_manager.cc:1905] T 26f70eac938142a88e3f455efee5628e P 9dad766957e545bf89d81dc1f7d9bb12: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 20:09:22.889328 6339 ts_tablet_manager.cc:1918] T 26f70eac938142a88e3f455efee5628e P 2637c8f627454dd78eff0b791765a030: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 20:09:22.889554 6339 log.cc:1196] T 26f70eac938142a88e3f455efee5628e P 2637c8f627454dd78eff0b791765a030: Deleting WAL directory at /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790194121577027-1706-0/minicluster-data/ts-1/wal/wals/26f70eac938142a88e3f455efee5628e
I20260923 20:09:22.890245 6339 ts_tablet_manager.cc:1939] T 26f70eac938142a88e3f455efee5628e P 2637c8f627454dd78eff0b791765a030: Deleting consensus metadata
I20260923 20:09:22.890939 6337 ts_tablet_manager.cc:1918] T 26f70eac938142a88e3f455efee5628e P 50ae4ca35842404aa6740bb97b073195: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 20:09:22.891110 6337 log.cc:1196] T 26f70eac938142a88e3f455efee5628e P 50ae4ca35842404aa6740bb97b073195: Deleting WAL directory at /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790194121577027-1706-0/minicluster-data/ts-0/wal/wals/26f70eac938142a88e3f455efee5628e
I20260923 20:09:22.891672 6337 ts_tablet_manager.cc:1939] T 26f70eac938142a88e3f455efee5628e P 50ae4ca35842404aa6740bb97b073195: Deleting consensus metadata
I20260923 20:09:22.892403 5827 catalog_manager.cc:4841] TS 2637c8f627454dd78eff0b791765a030 (127.1.170.130:45517): tablet 26f70eac938142a88e3f455efee5628e (table test-table [id=1b099ea8a0b94292a88ccee050c4bfca]) successfully deleted
I20260923 20:09:22.893286 5828 catalog_manager.cc:4841] TS 50ae4ca35842404aa6740bb97b073195 (127.1.170.129:34455): tablet 26f70eac938142a88e3f455efee5628e (table test-table [id=1b099ea8a0b94292a88ccee050c4bfca]) successfully deleted
Sep 23 20:09:22 dist-test-slave-th4g krb5kdc[5797](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194160, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.170.129@KRBTEST.COM
I20260923 20:09:22.897339 6338 ts_tablet_manager.cc:1918] T 26f70eac938142a88e3f455efee5628e P 9dad766957e545bf89d81dc1f7d9bb12: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 20:09:22.897516 6338 log.cc:1196] T 26f70eac938142a88e3f455efee5628e P 9dad766957e545bf89d81dc1f7d9bb12: Deleting WAL directory at /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790194121577027-1706-0/minicluster-data/ts-2/wal/wals/26f70eac938142a88e3f455efee5628e
I20260923 20:09:22.898048 6338 ts_tablet_manager.cc:1939] T 26f70eac938142a88e3f455efee5628e P 9dad766957e545bf89d81dc1f7d9bb12: Deleting consensus metadata
I20260923 20:09:22.899494 5827 catalog_manager.cc:4841] TS 9dad766957e545bf89d81dc1f7d9bb12 (127.1.170.131:35633): tablet 26f70eac938142a88e3f455efee5628e (table test-table [id=1b099ea8a0b94292a88ccee050c4bfca]) successfully deleted
I20260923 20:09:22.904336 6308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:09:22.869005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38176 (local address 127.1.170.129:34455)
0923 20:09:22.869221 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0923 20:09:22.869239 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0923 20:09:22.884229 (+ 14990us) server_negotiation.cc:407] Connection header received
0923 20:09:22.884486 (+ 257us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:09:22.884494 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:09:22.884593 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:09:22.884677 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:09:22.886467 (+ 1790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:09:22.887651 (+ 1184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:09:22.889499 (+ 1848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:09:22.890212 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:09:22.890307 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:09:22.897096 (+ 6789us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:09:22.897112 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:09:22.897132 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:09:22.897182 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:09:22.901274 (+ 4092us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:09:22.902052 (+ 778us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:09:22.902061 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:09:22.902065 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:09:22.902251 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:09:22.902609 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:09:22.902614 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:09:22.902617 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:09:22.903035 (+ 418us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:09:22.903323 (+ 288us) server_negotiation.cc:1035] Waiting for connection context
0923 20:09:22.903671 (+ 348us) server_negotiation.cc:299] Negotiation successful
0923 20:09:22.903929 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260923 20:09:22.905095 6024 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:09:23.004256 1706 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqUuku6/build/asan/bin/kudu with pid 5902
I20260923 20:09:23.021461 1706 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskqUuku6/build/asan/bin/kudu pid 6035
Sep 23 20:09:23 dist-test-slave-th4g krb5kdc[5797](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194160, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.170.130@KRBTEST.COM
I20260923 20:09:23.036963 6310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:09:23.022224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39842 (local address 127.1.170.130:45517)
0923 20:09:23.022400 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0923 20:09:23.022408 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 20:09:23.022492 (+ 84us) server_negotiation.cc:407] Connection header received
0923 20:09:23.022703 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:09:23.022709 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:09:23.022786 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:09:23.022887 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:09:23.024178 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:09:23.024897 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:09:23.026235 (+ 1338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:09:23.026619 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:09:23.026746 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:09:23.031267 (+ 4521us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:09:23.031281 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:09:23.031298 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:09:23.031335 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:09:23.034206 (+ 2871us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:09:23.034915 (+ 709us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:09:23.034923 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:09:23.034925 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:09:23.035046 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:09:23.035407 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:09:23.035414 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:09:23.035418 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:09:23.035784 (+ 366us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:09:23.036076 (+ 292us) server_negotiation.cc:1035] Waiting for connection context
0923 20:09:23.036365 (+ 289us) server_negotiation.cc:299] Negotiation successful
0923 20:09:23.036622 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260923 20:09:23.037685 6157 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:09:23.126698 1706 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqUuku6/build/asan/bin/kudu with pid 6035
I20260923 20:09:23.143288 1706 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskqUuku6/build/asan/bin/kudu pid 6168
Sep 23 20:09:23 dist-test-slave-th4g krb5kdc[5797](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194160, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.170.131@KRBTEST.COM
I20260923 20:09:23.157321 6312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:09:23.144028 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60956 (local address 127.1.170.131:35633)
0923 20:09:23.144220 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0923 20:09:23.144226 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:09:23.144256 (+ 30us) server_negotiation.cc:407] Connection header received
0923 20:09:23.144340 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:09:23.144343 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:09:23.144418 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:09:23.144518 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:09:23.145692 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:09:23.146458 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:09:23.147673 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:09:23.148048 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:09:23.148169 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:09:23.152228 (+ 4059us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:09:23.152248 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:09:23.152264 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:09:23.152296 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:09:23.154791 (+ 2495us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:09:23.155398 (+ 607us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:09:23.155404 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:09:23.155407 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:09:23.155532 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:09:23.155882 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:09:23.155889 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:09:23.155892 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:09:23.156208 (+ 316us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:09:23.156498 (+ 290us) server_negotiation.cc:1035] Waiting for connection context
0923 20:09:23.156793 (+ 295us) server_negotiation.cc:299] Negotiation successful
0923 20:09:23.157015 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260923 20:09:23.158056 6290 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:09:23.244901 1706 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqUuku6/build/asan/bin/kudu with pid 6168
I20260923 20:09:23.261216 1706 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskqUuku6/build/asan/bin/kudu pid 5810
I20260923 20:09:23.262648 5871 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:09:23.370388 1706 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqUuku6/build/asan/bin/kudu with pid 5810
2026-09-23T20:09:23Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (4291 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-09-23T20:09:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T20:09:23Z Disabled control of system clock
I20260923 20:09:23.436941 1706 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqUuku6/build/asan/bin/kudu
/tmp/dist-test-taskqUuku6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790194121577027-1706-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790194121577027-1706-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790194121577027-1706-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790194121577027-1706-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.170.190:34991
--webserver_interface=127.1.170.190
--webserver_port=0
--builtin_ntp_servers=127.1.170.148:40221
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.170.190:34991
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260923 20:09:23.754382 6348 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:09:23.754666 6348 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:09:23.754747 6348 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:09:23.764631 6348 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 20:09:23.764735 6348 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:09:23.764773 6348 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:09:23.764835 6348 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 20:09:23.764878 6348 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 20:09:23.773944 6348 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.170.148:40221
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790194121577027-1706-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790194121577027-1706-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.170.190:34991
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.170.190:34991
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790194121577027-1706-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.170.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790194121577027-1706-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:09:23.775116 6348 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:09:23.776726 6348 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:09:23.788043 6354 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:09:23.788079 6356 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:09:23.788137 6353 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:09:23.791509 6348 server_base.cc:1034] running on GCE node
I20260923 20:09:23.792416 6348 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:09:23.793936 6348 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:09:23.795184 6348 hybrid_clock.cc:648] HybridClock initialized: now 1790194163795113 us; error 34 us; skew 500 ppm
I20260923 20:09:23.795519 6348 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260923 20:09:23.798012 6348 webserver.cc:460] Webserver started at http://127.1.170.190:35285/ using document root <none> and password file <none>
I20260923 20:09:23.798710 6348 fs_manager.cc:359] Metadata directory not provided
I20260923 20:09:23.798803 6348 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:09:23.799067 6348 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:09:23.801860 6348 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790194121577027-1706-0/minicluster-data/master-0/data/instance:
uuid: "f3fc7212a9c1456da161d7c108ad1f75"
format_stamp: "Formatted at 2026-09-23 20:09:23 on dist-test-slave-th4g"
server_key: "4c2c352fdba37a54c17c3094073442eb"
server_key_iv: "0b23557e606b670f6034cbbecf812f53"
server_key_version: "encryptionkey@0"
I20260923 20:09:23.802675 6348 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790194121577027-1706-0/minicluster-data/master-0/wal/instance:
uuid: "f3fc7212a9c1456da161d7c108ad1f75"
format_stamp: "Formatted at 2026-09-23 20:09:23 on dist-test-slave-th4g"
server_key: "4c2c352fdba37a54c17c3094073442eb"
server_key_iv: "0b23557e606b670f6034cbbecf812f53"
server_key_version: "encryptionkey@0"
I20260923 20:09:23.807689 6348 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260923 20:09:23.811060 6362 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:09:23.812584 6348 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260923 20:09:23.812734 6348 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790194121577027-1706-0/minicluster-data/master-0/data,/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790194121577027-1706-0/minicluster-data/master-0/wal
uuid: "f3fc7212a9c1456da161d7c108ad1f75"
format_stamp: "Formatted at 2026-09-23 20:09:23 on dist-test-slave-th4g"
server_key: "4c2c352fdba37a54c17c3094073442eb"
server_key_iv: "0b23557e606b670f6034cbbecf812f53"
server_key_version: "encryptionkey@0"
I20260923 20:09:23.812911 6348 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790194121577027-1706-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790194121577027-1706-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790194121577027-1706-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:09:23.818356 6348 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:09:23.819036 6348 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:09:23.819291 6348 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:09:23.837038 6348 rpc_server.cc:307] RPC server started. Bound to: 127.1.170.190:34991
I20260923 20:09:23.837080 6413 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.170.190:34991 every 8 connection(s)
I20260923 20:09:23.838388 6348 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790194121577027-1706-0/minicluster-data/master-0/data/info.pb
I20260923 20:09:23.841893 6414 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:09:23.845854 1706 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqUuku6/build/asan/bin/kudu as pid 6348
I20260923 20:09:23.846217 1706 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790194121577027-1706-0/minicluster-data/master-0/wal/instance
I20260923 20:09:23.846817 1706 external_mini_cluster.cc:1428] Setting key 66061f05f189507eeb561abe2d1e68c1
I20260923 20:09:23.852392 6414 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f3fc7212a9c1456da161d7c108ad1f75: Bootstrap starting.
I20260923 20:09:23.855209 6417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:09:23.848780 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44890 (local address 127.1.170.190:34991)
0923 20:09:23.849819 (+ 1039us) server_negotiation.cc:206] Beginning negotiation
0923 20:09:23.849840 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0923 20:09:23.849942 (+ 102us) server_negotiation.cc:407] Connection header received
0923 20:09:23.851810 (+ 1868us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:09:23.851832 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:09:23.852225 (+ 393us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:09:23.852596 (+ 371us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:09:23.853650 (+ 1054us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:09:23.853656 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:09:23.853669 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 20:09:23.853689 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:09:23.853748 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 20:09:23.853763 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:09:23.854021 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0923 20:09:23.854155 (+ 134us) server_negotiation.cc:299] Negotiation successful
0923 20:09:23.854395 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":631,"thread_start_us":392,"threads_started":1}
I20260923 20:09:23.855775 6414 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f3fc7212a9c1456da161d7c108ad1f75: Neither blocks nor log segments found. Creating new log.
I20260923 20:09:23.856923 6414 log.cc:824] T 00000000000000000000000000000000 P f3fc7212a9c1456da161d7c108ad1f75: Log is configured to *not* fsync() on all Append() calls
I20260923 20:09:23.859722 6414 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f3fc7212a9c1456da161d7c108ad1f75: No bootstrap required, opened a new log
I20260923 20:09:23.866717 6414 raft_consensus.cc:348] T 00000000000000000000000000000000 P f3fc7212a9c1456da161d7c108ad1f75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3fc7212a9c1456da161d7c108ad1f75" member_type: VOTER last_known_addr { host: "127.1.170.190" port: 34991 } }
I20260923 20:09:23.867053 6414 raft_consensus.cc:374] T 00000000000000000000000000000000 P f3fc7212a9c1456da161d7c108ad1f75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:09:23.867161 6414 raft_consensus.cc:729] T 00000000000000000000000000000000 P f3fc7212a9c1456da161d7c108ad1f75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f3fc7212a9c1456da161d7c108ad1f75, State: Initialized, Role: FOLLOWER
I20260923 20:09:23.867700 6414 consensus_queue.cc:260] T 00000000000000000000000000000000 P f3fc7212a9c1456da161d7c108ad1f75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3fc7212a9c1456da161d7c108ad1f75" member_type: VOTER last_known_addr { host: "127.1.170.190" port: 34991 } }
I20260923 20:09:23.867913 6414 raft_consensus.cc:388] T 00000000000000000000000000000000 P f3fc7212a9c1456da161d7c108ad1f75 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 20:09:23.868093 6414 raft_consensus.cc:482] T 00000000000000000000000000000000 P f3fc7212a9c1456da161d7c108ad1f75 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 20:09:23.868290 6414 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f3fc7212a9c1456da161d7c108ad1f75 [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:09:23.870603 6414 raft_consensus.cc:504] T 00000000000000000000000000000000 P f3fc7212a9c1456da161d7c108ad1f75 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3fc7212a9c1456da161d7c108ad1f75" member_type: VOTER last_known_addr { host: "127.1.170.190" port: 34991 } }
I20260923 20:09:23.871269 6414 leader_election.cc:302] T 00000000000000000000000000000000 P f3fc7212a9c1456da161d7c108ad1f75 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f3fc7212a9c1456da161d7c108ad1f75; no voters:
I20260923 20:09:23.871883 6414 leader_election.cc:288] T 00000000000000000000000000000000 P f3fc7212a9c1456da161d7c108ad1f75 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 20:09:23.872027 6419 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f3fc7212a9c1456da161d7c108ad1f75 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 20:09:23.873025 6419 raft_consensus.cc:686] T 00000000000000000000000000000000 P f3fc7212a9c1456da161d7c108ad1f75 [term 1 LEADER]: Becoming Leader. State: Replica: f3fc7212a9c1456da161d7c108ad1f75, State: Running, Role: LEADER
I20260923 20:09:23.873543 6419 consensus_queue.cc:237] T 00000000000000000000000000000000 P f3fc7212a9c1456da161d7c108ad1f75 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3fc7212a9c1456da161d7c108ad1f75" member_type: VOTER last_known_addr { host: "127.1.170.190" port: 34991 } }
I20260923 20:09:23.873852 6414 sys_catalog.cc:564] T 00000000000000000000000000000000 P f3fc7212a9c1456da161d7c108ad1f75 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 20:09:23.876312 6421 sys_catalog.cc:455] T 00000000000000000000000000000000 P f3fc7212a9c1456da161d7c108ad1f75 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f3fc7212a9c1456da161d7c108ad1f75. Latest consensus state: current_term: 1 leader_uuid: "f3fc7212a9c1456da161d7c108ad1f75" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3fc7212a9c1456da161d7c108ad1f75" member_type: VOTER last_known_addr { host: "127.1.170.190" port: 34991 } } }
I20260923 20:09:23.876415 6420 sys_catalog.cc:455] T 00000000000000000000000000000000 P f3fc7212a9c1456da161d7c108ad1f75 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f3fc7212a9c1456da161d7c108ad1f75" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3fc7212a9c1456da161d7c108ad1f75" member_type: VOTER last_known_addr { host: "127.1.170.190" port: 34991 } } }
I20260923 20:09:23.876704 6421 sys_catalog.cc:458] T 00000000000000000000000000000000 P f3fc7212a9c1456da161d7c108ad1f75 [sys.catalog]: This master's current role is: LEADER
I20260923 20:09:23.876703 6420 sys_catalog.cc:458] T 00000000000000000000000000000000 P f3fc7212a9c1456da161d7c108ad1f75 [sys.catalog]: This master's current role is: LEADER
I20260923 20:09:23.880904 6427 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 20:09:23.885075 6427 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 20:09:23.892532 6427 catalog_manager.cc:1347] Generated new cluster ID: 608ba52367b745fdb02fde601ce9cddc
I20260923 20:09:23.892655 6427 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 20:09:23.913417 6427 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 20:09:23.913862 6427 catalog_manager.cc:1504] Loading token signing keys...
I20260923 20:09:23.923182 6427 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f3fc7212a9c1456da161d7c108ad1f75: Generated new TSK 0
I20260923 20:09:23.924057 6427 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 20:09:23.930861 1706 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqUuku6/build/asan/bin/kudu
/tmp/dist-test-taskqUuku6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790194121577027-1706-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790194121577027-1706-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790194121577027-1706-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790194121577027-1706-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.170.129:0
--local_ip_for_outbound_sockets=127.1.170.129
--webserver_interface=127.1.170.129
--webserver_port=0
--tserver_master_addrs=127.1.170.190:34991
--builtin_ntp_servers=127.1.170.148:40221
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260923 20:09:24.256501 6438 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:09:24.256809 6438 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:09:24.257000 6438 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:09:24.264727 6438 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:09:24.264846 6438 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:09:24.265038 6438 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.170.129
I20260923 20:09:24.276257 6438 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.170.148:40221
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790194121577027-1706-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790194121577027-1706-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.170.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790194121577027-1706-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.170.129
--webserver_port=0
--tserver_master_addrs=127.1.170.190:34991
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.170.129
--log_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790194121577027-1706-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:09:24.277570 6438 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:09:24.279436 6438 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:09:24.292555 6443 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:09:24.292459 6446 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:09:24.292601 6444 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:09:24.293358 6438 server_base.cc:1034] running on GCE node
I20260923 20:09:24.293948 6438 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:09:24.294874 6438 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:09:24.296092 6438 hybrid_clock.cc:648] HybridClock initialized: now 1790194164296045 us; error 38 us; skew 500 ppm
I20260923 20:09:24.296440 6438 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260923 20:09:24.304935 6438 webserver.cc:460] Webserver started at http://127.1.170.129:40677/ using document root <none> and password file <none>
I20260923 20:09:24.305775 6438 fs_manager.cc:359] Metadata directory not provided
I20260923 20:09:24.305972 6438 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:09:24.306351 6438 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:09:24.309280 6438 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790194121577027-1706-0/minicluster-data/ts-0/data/instance:
uuid: "ea61b75cc55742aaa8987ef1f487979e"
format_stamp: "Formatted at 2026-09-23 20:09:24 on dist-test-slave-th4g"
server_key: "428a7734fe35a8c723e79448dcaa17d1"
server_key_iv: "b13edfe24aad59aeff16fec74506b2af"
server_key_version: "encryptionkey@0"
I20260923 20:09:24.310109 6438 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790194121577027-1706-0/minicluster-data/ts-0/wal/instance:
uuid: "ea61b75cc55742aaa8987ef1f487979e"
format_stamp: "Formatted at 2026-09-23 20:09:24 on dist-test-slave-th4g"
server_key: "428a7734fe35a8c723e79448dcaa17d1"
server_key_iv: "b13edfe24aad59aeff16fec74506b2af"
server_key_version: "encryptionkey@0"
I20260923 20:09:24.315272 6438 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260923 20:09:24.318938 6452 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:09:24.320382 6438 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260923 20:09:24.320523 6438 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790194121577027-1706-0/minicluster-data/ts-0/data,/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790194121577027-1706-0/minicluster-data/ts-0/wal
uuid: "ea61b75cc55742aaa8987ef1f487979e"
format_stamp: "Formatted at 2026-09-23 20:09:24 on dist-test-slave-th4g"
server_key: "428a7734fe35a8c723e79448dcaa17d1"
server_key_iv: "b13edfe24aad59aeff16fec74506b2af"
server_key_version: "encryptionkey@0"
I20260923 20:09:24.320745 6438 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790194121577027-1706-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790194121577027-1706-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790194121577027-1706-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:09:24.327080 6438 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:09:24.327868 6438 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:09:24.328104 6438 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:09:24.328982 6438 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 20:09:24.330461 6438 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 20:09:24.330545 6438 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 20:09:24.330629 6438 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 20:09:24.330683 6438 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 20:09:24.365549 6438 rpc_server.cc:307] RPC server started. Bound to: 127.1.170.129:40663
I20260923 20:09:24.365643 6564 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.170.129:40663 every 8 connection(s)
I20260923 20:09:24.366920 6438 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790194121577027-1706-0/minicluster-data/ts-0/data/info.pb
I20260923 20:09:24.370896 1706 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqUuku6/build/asan/bin/kudu as pid 6438
I20260923 20:09:24.371135 1706 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790194121577027-1706-0/minicluster-data/ts-0/wal/instance
I20260923 20:09:24.371834 1706 external_mini_cluster.cc:1428] Setting key 68a05d1ed41f82ed09cdbe62f6803dfb
I20260923 20:09:24.374691 1706 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqUuku6/build/asan/bin/kudu
/tmp/dist-test-taskqUuku6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790194121577027-1706-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790194121577027-1706-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790194121577027-1706-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790194121577027-1706-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.170.130:0
--local_ip_for_outbound_sockets=127.1.170.130
--webserver_interface=127.1.170.130
--webserver_port=0
--tserver_master_addrs=127.1.170.190:34991
--builtin_ntp_servers=127.1.170.148:40221
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260923 20:09:24.376092 6567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:09:24.369608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.170.129:55321 (local address 127.1.170.190:34991)
0923 20:09:24.370250 (+ 642us) server_negotiation.cc:206] Beginning negotiation
0923 20:09:24.370255 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 20:09:24.371807 (+ 1552us) server_negotiation.cc:407] Connection header received
0923 20:09:24.373450 (+ 1643us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:09:24.373456 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:09:24.373525 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:09:24.373638 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:09:24.375273 (+ 1635us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:09:24.375280 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:09:24.375288 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 20:09:24.375305 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:09:24.375416 (+ 111us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 20:09:24.375431 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:09:24.375562 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0923 20:09:24.375759 (+ 197us) server_negotiation.cc:299] Negotiation successful
0923 20:09:24.375826 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":533,"thread_start_us":451,"threads_started":1}
I20260923 20:09:24.377077 6568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:09:24.370535 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.190:34991 (local address 127.1.170.129:55321)
0923 20:09:24.371609 (+ 1074us) negotiation.cc:107] Waiting for socket to connect
0923 20:09:24.371659 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0923 20:09:24.373180 (+ 1521us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:09:24.373933 (+ 753us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:09:24.373945 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:09:24.374052 (+ 107us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:09:24.374865 (+ 813us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 20:09:24.374871 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:09:24.374931 (+ 60us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 20:09:24.375056 (+ 125us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:09:24.375531 (+ 475us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:09:24.375543 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:09:24.375552 (+ 9us) client_negotiation.cc:769] Sending connection context
0923 20:09:24.375874 (+ 322us) client_negotiation.cc:240] Negotiation successful
0923 20:09:24.376140 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":552,"thread_start_us":371,"threads_started":1}
I20260923 20:09:24.378804 6565 heartbeater.cc:347] Connected to a master server at 127.1.170.190:34991
I20260923 20:09:24.379153 6565 heartbeater.cc:464] Registering TS with master...
I20260923 20:09:24.379896 6565 heartbeater.cc:511] Master 127.1.170.190:34991 requested a full tablet report, sending...
I20260923 20:09:24.382213 6379 ts_manager.cc:194] Registered new tserver with Master: ea61b75cc55742aaa8987ef1f487979e (127.1.170.129:40663)
W20260923 20:09:24.701395 6569 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:09:24.701730 6569 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:09:24.701936 6569 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:09:24.709434 6569 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:09:24.709583 6569 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:09:24.709764 6569 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.170.130
I20260923 20:09:24.719365 6569 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.170.148:40221
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790194121577027-1706-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790194121577027-1706-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.170.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790194121577027-1706-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.170.130
--webserver_port=0
--tserver_master_addrs=127.1.170.190:34991
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.170.130
--log_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790194121577027-1706-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:09:24.720691 6569 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:09:24.722446 6569 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:09:24.735296 6574 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:09:24.735363 6575 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:09:24.735330 6577 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:09:24.736202 6569 server_base.cc:1034] running on GCE node
I20260923 20:09:24.736795 6569 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:09:24.737792 6569 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:09:24.739025 6569 hybrid_clock.cc:648] HybridClock initialized: now 1790194164738969 us; error 49 us; skew 500 ppm
I20260923 20:09:24.739367 6569 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260923 20:09:24.741710 6569 webserver.cc:460] Webserver started at http://127.1.170.130:34345/ using document root <none> and password file <none>
I20260923 20:09:24.742578 6569 fs_manager.cc:359] Metadata directory not provided
I20260923 20:09:24.742726 6569 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:09:24.743050 6569 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:09:24.745994 6569 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790194121577027-1706-0/minicluster-data/ts-1/data/instance:
uuid: "e087226ecd1b461c823911514d96bf69"
format_stamp: "Formatted at 2026-09-23 20:09:24 on dist-test-slave-th4g"
server_key: "62d940409d11e31670f73d9319918668"
server_key_iv: "38850f83f5f19a5f5237800604c632df"
server_key_version: "encryptionkey@0"
I20260923 20:09:24.746803 6569 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790194121577027-1706-0/minicluster-data/ts-1/wal/instance:
uuid: "e087226ecd1b461c823911514d96bf69"
format_stamp: "Formatted at 2026-09-23 20:09:24 on dist-test-slave-th4g"
server_key: "62d940409d11e31670f73d9319918668"
server_key_iv: "38850f83f5f19a5f5237800604c632df"
server_key_version: "encryptionkey@0"
I20260923 20:09:24.752136 6569 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260923 20:09:24.755887 6583 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:09:24.757195 6569 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260923 20:09:24.757392 6569 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790194121577027-1706-0/minicluster-data/ts-1/data,/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790194121577027-1706-0/minicluster-data/ts-1/wal
uuid: "e087226ecd1b461c823911514d96bf69"
format_stamp: "Formatted at 2026-09-23 20:09:24 on dist-test-slave-th4g"
server_key: "62d940409d11e31670f73d9319918668"
server_key_iv: "38850f83f5f19a5f5237800604c632df"
server_key_version: "encryptionkey@0"
I20260923 20:09:24.757627 6569 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790194121577027-1706-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790194121577027-1706-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790194121577027-1706-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:09:24.763425 6569 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:09:24.764298 6569 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:09:24.764582 6569 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:09:24.765573 6569 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 20:09:24.767087 6569 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 20:09:24.767225 6569 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 20:09:24.767359 6569 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 20:09:24.767445 6569 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 20:09:24.800117 6569 rpc_server.cc:307] RPC server started. Bound to: 127.1.170.130:36217
I20260923 20:09:24.800154 6695 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.170.130:36217 every 8 connection(s)
I20260923 20:09:24.801497 6569 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790194121577027-1706-0/minicluster-data/ts-1/data/info.pb
I20260923 20:09:24.806725 1706 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqUuku6/build/asan/bin/kudu as pid 6569
I20260923 20:09:24.806929 1706 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790194121577027-1706-0/minicluster-data/ts-1/wal/instance
I20260923 20:09:24.809044 1706 external_mini_cluster.cc:1428] Setting key 48f36a6ab73bc93c5add17b933bbac42
I20260923 20:09:24.812744 1706 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqUuku6/build/asan/bin/kudu
/tmp/dist-test-taskqUuku6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790194121577027-1706-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790194121577027-1706-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790194121577027-1706-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790194121577027-1706-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.170.131:0
--local_ip_for_outbound_sockets=127.1.170.131
--webserver_interface=127.1.170.131
--webserver_port=0
--tserver_master_addrs=127.1.170.190:34991
--builtin_ntp_servers=127.1.170.148:40221
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260923 20:09:24.813396 6567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:09:24.804478 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.170.130:59489 (local address 127.1.170.190:34991)
0923 20:09:24.804670 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0923 20:09:24.804675 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 20:09:24.806115 (+ 1440us) server_negotiation.cc:407] Connection header received
0923 20:09:24.810803 (+ 4688us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:09:24.810809 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:09:24.810882 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:09:24.811010 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:09:24.812646 (+ 1636us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:09:24.812657 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:09:24.812662 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 20:09:24.812686 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:09:24.812725 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 20:09:24.812739 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:09:24.812883 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0923 20:09:24.813127 (+ 244us) server_negotiation.cc:299] Negotiation successful
0923 20:09:24.813184 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260923 20:09:24.815011 6698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:09:24.804945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.190:34991 (local address 127.1.170.130:59489)
0923 20:09:24.805958 (+ 1013us) negotiation.cc:107] Waiting for socket to connect
0923 20:09:24.805998 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0923 20:09:24.810427 (+ 4429us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:09:24.811287 (+ 860us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:09:24.811297 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:09:24.811407 (+ 110us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:09:24.812188 (+ 781us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 20:09:24.812192 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:09:24.812249 (+ 57us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 20:09:24.812385 (+ 136us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:09:24.812866 (+ 481us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:09:24.812878 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:09:24.812890 (+ 12us) client_negotiation.cc:769] Sending connection context
0923 20:09:24.813226 (+ 336us) client_negotiation.cc:240] Negotiation successful
0923 20:09:24.813540 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":467,"thread_start_us":273,"threads_started":1}
I20260923 20:09:24.816512 6696 heartbeater.cc:347] Connected to a master server at 127.1.170.190:34991
I20260923 20:09:24.816886 6696 heartbeater.cc:464] Registering TS with master...
I20260923 20:09:24.817668 6696 heartbeater.cc:511] Master 127.1.170.190:34991 requested a full tablet report, sending...
I20260923 20:09:24.819380 6379 ts_manager.cc:194] Registered new tserver with Master: e087226ecd1b461c823911514d96bf69 (127.1.170.130:36217)
W20260923 20:09:25.148888 6699 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:09:25.149224 6699 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:09:25.149374 6699 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:09:25.157076 6699 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:09:25.157219 6699 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:09:25.157392 6699 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.170.131
I20260923 20:09:25.166841 6699 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.170.148:40221
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790194121577027-1706-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790194121577027-1706-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.170.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790194121577027-1706-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.170.131
--webserver_port=0
--tserver_master_addrs=127.1.170.190:34991
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.170.131
--log_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790194121577027-1706-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:09:25.168190 6699 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:09:25.169992 6699 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:09:25.183501 6704 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:09:25.183581 6707 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:09:25.183554 6705 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:09:25.184846 6699 server_base.cc:1034] running on GCE node
I20260923 20:09:25.185523 6699 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:09:25.186599 6699 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:09:25.187875 6699 hybrid_clock.cc:648] HybridClock initialized: now 1790194165187823 us; error 42 us; skew 500 ppm
I20260923 20:09:25.188235 6699 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260923 20:09:25.192278 6699 webserver.cc:460] Webserver started at http://127.1.170.131:36759/ using document root <none> and password file <none>
I20260923 20:09:25.193470 6699 fs_manager.cc:359] Metadata directory not provided
I20260923 20:09:25.193648 6699 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:09:25.194088 6699 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:09:25.198150 6699 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790194121577027-1706-0/minicluster-data/ts-2/data/instance:
uuid: "50602caf24974536bb98d52fc5d2ab3f"
format_stamp: "Formatted at 2026-09-23 20:09:25 on dist-test-slave-th4g"
server_key: "d339eeeff0a220424532c0574eb2308f"
server_key_iv: "e43109d58210a3b779ae2cf6bbb27585"
server_key_version: "encryptionkey@0"
I20260923 20:09:25.199190 6699 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790194121577027-1706-0/minicluster-data/ts-2/wal/instance:
uuid: "50602caf24974536bb98d52fc5d2ab3f"
format_stamp: "Formatted at 2026-09-23 20:09:25 on dist-test-slave-th4g"
server_key: "d339eeeff0a220424532c0574eb2308f"
server_key_iv: "e43109d58210a3b779ae2cf6bbb27585"
server_key_version: "encryptionkey@0"
I20260923 20:09:25.208870 6699 fs_manager.cc:691] Time spent creating directory manager: real 0.009s user 0.005s sys 0.000s
I20260923 20:09:25.215184 6713 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:09:25.218941 6699 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.003s sys 0.002s
I20260923 20:09:25.219247 6699 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790194121577027-1706-0/minicluster-data/ts-2/data,/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790194121577027-1706-0/minicluster-data/ts-2/wal
uuid: "50602caf24974536bb98d52fc5d2ab3f"
format_stamp: "Formatted at 2026-09-23 20:09:25 on dist-test-slave-th4g"
server_key: "d339eeeff0a220424532c0574eb2308f"
server_key_iv: "e43109d58210a3b779ae2cf6bbb27585"
server_key_version: "encryptionkey@0"
I20260923 20:09:25.219682 6699 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790194121577027-1706-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790194121577027-1706-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790194121577027-1706-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:09:25.229003 6699 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:09:25.230271 6699 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:09:25.230674 6699 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:09:25.232043 6699 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 20:09:25.234465 6699 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 20:09:25.234683 6699 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.001s
I20260923 20:09:25.234876 6699 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 20:09:25.235023 6699 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 20:09:25.279291 6699 rpc_server.cc:307] RPC server started. Bound to: 127.1.170.131:33747
I20260923 20:09:25.279327 6825 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.170.131:33747 every 8 connection(s)
I20260923 20:09:25.281484 6699 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790194121577027-1706-0/minicluster-data/ts-2/data/info.pb
I20260923 20:09:25.285703 1706 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqUuku6/build/asan/bin/kudu as pid 6699
I20260923 20:09:25.285966 1706 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790194121577027-1706-0/minicluster-data/ts-2/wal/instance
I20260923 20:09:25.286531 1706 external_mini_cluster.cc:1428] Setting key f913c4c5da880a686f18ea7d64981aa5
I20260923 20:09:25.292763 6567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:09:25.284893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.170.131:41857 (local address 127.1.170.190:34991)
0923 20:09:25.285110 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0923 20:09:25.285117 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:09:25.287118 (+ 2001us) server_negotiation.cc:407] Connection header received
0923 20:09:25.289911 (+ 2793us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:09:25.289917 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:09:25.289987 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:09:25.290096 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:09:25.291821 (+ 1725us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:09:25.291825 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:09:25.291829 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 20:09:25.291845 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:09:25.291876 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 20:09:25.291888 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:09:25.291991 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0923 20:09:25.292288 (+ 297us) server_negotiation.cc:299] Negotiation successful
0923 20:09:25.292368 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260923 20:09:25.295015 6828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:09:25.285608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.190:34991 (local address 127.1.170.131:41857)
0923 20:09:25.286940 (+ 1332us) negotiation.cc:107] Waiting for socket to connect
0923 20:09:25.286995 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0923 20:09:25.289582 (+ 2587us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:09:25.290373 (+ 791us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:09:25.290389 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:09:25.290535 (+ 146us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:09:25.291384 (+ 849us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 20:09:25.291389 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:09:25.291447 (+ 58us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 20:09:25.291577 (+ 130us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:09:25.292003 (+ 426us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:09:25.292018 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:09:25.292027 (+ 9us) client_negotiation.cc:769] Sending connection context
0923 20:09:25.293141 (+ 1114us) client_negotiation.cc:240] Negotiation successful
0923 20:09:25.293479 (+ 338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":714,"thread_start_us":444,"threads_started":1}
I20260923 20:09:25.296487 6826 heartbeater.cc:347] Connected to a master server at 127.1.170.190:34991
I20260923 20:09:25.296823 6826 heartbeater.cc:464] Registering TS with master...
I20260923 20:09:25.297428 6826 heartbeater.cc:511] Master 127.1.170.190:34991 requested a full tablet report, sending...
I20260923 20:09:25.298895 6379 ts_manager.cc:194] Registered new tserver with Master: 50602caf24974536bb98d52fc5d2ab3f (127.1.170.131:33747)
I20260923 20:09:25.302762 1706 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260923 20:09:25.310106 6567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:09:25.307413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44892 (local address 127.1.170.190:34991)
0923 20:09:25.307587 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0923 20:09:25.307591 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 20:09:25.308041 (+ 450us) server_negotiation.cc:407] Connection header received
0923 20:09:25.308170 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:09:25.308174 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:09:25.308246 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:09:25.308338 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:09:25.309369 (+ 1031us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:09:25.309374 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:09:25.309377 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 20:09:25.309392 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:09:25.309426 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 20:09:25.309437 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:09:25.309543 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0923 20:09:25.309746 (+ 203us) server_negotiation.cc:299] Negotiation successful
0923 20:09:25.309797 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260923 20:09:25.314833 6379 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44892:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 20:09:25.321771 6379 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 20:09:25.341181 6840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:09:25.334713 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46324 (local address 127.1.170.129:40663)
0923 20:09:25.337155 (+ 2442us) server_negotiation.cc:206] Beginning negotiation
0923 20:09:25.337161 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:09:25.337186 (+ 25us) server_negotiation.cc:407] Connection header received
0923 20:09:25.337245 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:09:25.337251 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:09:25.337512 (+ 261us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:09:25.337679 (+ 167us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:09:25.340300 (+ 2621us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:09:25.340308 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:09:25.340323 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 20:09:25.340343 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:09:25.340402 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 20:09:25.340428 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:09:25.340561 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0923 20:09:25.340697 (+ 136us) server_negotiation.cc:299] Negotiation successful
0923 20:09:25.340815 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2270,"thread_start_us":2159,"threads_started":1}
I20260923 20:09:25.341285 6839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:09:25.334552 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.129:40663 (local address 127.0.0.1:46324)
0923 20:09:25.336397 (+ 1845us) negotiation.cc:107] Waiting for socket to connect
0923 20:09:25.336453 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0923 20:09:25.336639 (+ 186us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:09:25.338914 (+ 2275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:09:25.338925 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:09:25.339051 (+ 126us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:09:25.340022 (+ 971us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 20:09:25.340025 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:09:25.340060 (+ 35us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 20:09:25.340140 (+ 80us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:09:25.340530 (+ 390us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:09:25.340539 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:09:25.340547 (+ 8us) client_negotiation.cc:769] Sending connection context
0923 20:09:25.340729 (+ 182us) client_negotiation.cc:240] Negotiation successful
0923 20:09:25.340812 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1514,"thread_start_us":1318,"threads_started":1}
I20260923 20:09:25.343822 6839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:09:25.334884 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.130:36217 (local address 127.0.0.1:32940)
0923 20:09:25.341503 (+ 6619us) negotiation.cc:107] Waiting for socket to connect
0923 20:09:25.341526 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0923 20:09:25.341616 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:09:25.342183 (+ 567us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:09:25.342187 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:09:25.342235 (+ 48us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:09:25.342860 (+ 625us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 20:09:25.342863 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:09:25.342879 (+ 16us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 20:09:25.342921 (+ 42us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:09:25.343404 (+ 483us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:09:25.343413 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:09:25.343421 (+ 8us) client_negotiation.cc:769] Sending connection context
0923 20:09:25.343515 (+ 94us) client_negotiation.cc:240] Negotiation successful
0923 20:09:25.343565 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6496,"threads_started":1}
I20260923 20:09:25.344628 6838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:09:25.334371 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32940 (local address 127.1.170.130:36217)
0923 20:09:25.334975 (+ 604us) server_negotiation.cc:206] Beginning negotiation
0923 20:09:25.334989 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0923 20:09:25.341729 (+ 6740us) server_negotiation.cc:407] Connection header received
0923 20:09:25.341792 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:09:25.341798 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:09:25.342035 (+ 237us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:09:25.342186 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:09:25.343107 (+ 921us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:09:25.343116 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:09:25.343128 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 20:09:25.343145 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:09:25.343193 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 20:09:25.343216 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:09:25.343349 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0923 20:09:25.344166 (+ 817us) server_negotiation.cc:299] Negotiation successful
0923 20:09:25.344274 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":406,"thread_start_us":282,"threads_started":1}
I20260923 20:09:25.346458 6842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:09:25.336900 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.131:33747 (local address 127.0.0.1:45190)
0923 20:09:25.342349 (+ 5449us) negotiation.cc:107] Waiting for socket to connect
0923 20:09:25.342371 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0923 20:09:25.343022 (+ 651us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:09:25.344608 (+ 1586us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:09:25.344615 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:09:25.344642 (+ 27us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:09:25.345381 (+ 739us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 20:09:25.345386 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:09:25.345402 (+ 16us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 20:09:25.345452 (+ 50us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:09:25.345980 (+ 528us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:09:25.345989 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:09:25.345993 (+ 4us) client_negotiation.cc:769] Sending connection context
0923 20:09:25.346124 (+ 131us) client_negotiation.cc:240] Negotiation successful
0923 20:09:25.346169 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5307,"thread_start_us":5157,"threads_started":1}
I20260923 20:09:25.347137 6843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:09:25.337476 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45190 (local address 127.1.170.131:33747)
0923 20:09:25.340448 (+ 2972us) server_negotiation.cc:206] Beginning negotiation
0923 20:09:25.340458 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0923 20:09:25.342456 (+ 1998us) server_negotiation.cc:407] Connection header received
0923 20:09:25.343165 (+ 709us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:09:25.343172 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:09:25.343361 (+ 189us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:09:25.343499 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:09:25.345609 (+ 2110us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:09:25.345616 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:09:25.345629 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 20:09:25.345645 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:09:25.345727 (+ 82us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 20:09:25.345748 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:09:25.345928 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0923 20:09:25.346730 (+ 802us) server_negotiation.cc:299] Negotiation successful
0923 20:09:25.346817 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2794,"thread_start_us":2414,"threads_started":1}
I20260923 20:09:25.348802 6631 tablet_service.cc:1467] Processing CreateTablet for tablet 41ec739902bf46f887f1ff8bb704120f (DEFAULT_TABLE table=test-table [id=fec47bb63d194e26ba128e3d8c421505]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 20:09:25.349745 6500 tablet_service.cc:1467] Processing CreateTablet for tablet 41ec739902bf46f887f1ff8bb704120f (DEFAULT_TABLE table=test-table [id=fec47bb63d194e26ba128e3d8c421505]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 20:09:25.349797 6761 tablet_service.cc:1467] Processing CreateTablet for tablet 41ec739902bf46f887f1ff8bb704120f (DEFAULT_TABLE table=test-table [id=fec47bb63d194e26ba128e3d8c421505]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 20:09:25.350384 6631 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 41ec739902bf46f887f1ff8bb704120f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:09:25.350970 6761 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 41ec739902bf46f887f1ff8bb704120f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:09:25.351102 6500 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 41ec739902bf46f887f1ff8bb704120f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:09:25.363479 6844 tablet_bootstrap.cc:492] T 41ec739902bf46f887f1ff8bb704120f P 50602caf24974536bb98d52fc5d2ab3f: Bootstrap starting.
I20260923 20:09:25.365715 6846 tablet_bootstrap.cc:492] T 41ec739902bf46f887f1ff8bb704120f P ea61b75cc55742aaa8987ef1f487979e: Bootstrap starting.
I20260923 20:09:25.365899 6845 tablet_bootstrap.cc:492] T 41ec739902bf46f887f1ff8bb704120f P e087226ecd1b461c823911514d96bf69: Bootstrap starting.
I20260923 20:09:25.367131 6844 tablet_bootstrap.cc:654] T 41ec739902bf46f887f1ff8bb704120f P 50602caf24974536bb98d52fc5d2ab3f: Neither blocks nor log segments found. Creating new log.
I20260923 20:09:25.369614 6846 tablet_bootstrap.cc:654] T 41ec739902bf46f887f1ff8bb704120f P ea61b75cc55742aaa8987ef1f487979e: Neither blocks nor log segments found. Creating new log.
I20260923 20:09:25.369840 6844 log.cc:824] T 41ec739902bf46f887f1ff8bb704120f P 50602caf24974536bb98d52fc5d2ab3f: Log is configured to *not* fsync() on all Append() calls
I20260923 20:09:25.370626 6845 tablet_bootstrap.cc:654] T 41ec739902bf46f887f1ff8bb704120f P e087226ecd1b461c823911514d96bf69: Neither blocks nor log segments found. Creating new log.
I20260923 20:09:25.371114 6846 log.cc:824] T 41ec739902bf46f887f1ff8bb704120f P ea61b75cc55742aaa8987ef1f487979e: Log is configured to *not* fsync() on all Append() calls
I20260923 20:09:25.371905 6845 log.cc:824] T 41ec739902bf46f887f1ff8bb704120f P e087226ecd1b461c823911514d96bf69: Log is configured to *not* fsync() on all Append() calls
I20260923 20:09:25.382126 6844 tablet_bootstrap.cc:492] T 41ec739902bf46f887f1ff8bb704120f P 50602caf24974536bb98d52fc5d2ab3f: No bootstrap required, opened a new log
I20260923 20:09:25.382643 6844 ts_tablet_manager.cc:1397] T 41ec739902bf46f887f1ff8bb704120f P 50602caf24974536bb98d52fc5d2ab3f: Time spent bootstrapping tablet: real 0.019s user 0.013s sys 0.002s
I20260923 20:09:25.385260 6565 heartbeater.cc:503] Master 127.1.170.190:34991 was elected leader, sending a full tablet report...
I20260923 20:09:25.385521 6846 tablet_bootstrap.cc:492] T 41ec739902bf46f887f1ff8bb704120f P ea61b75cc55742aaa8987ef1f487979e: No bootstrap required, opened a new log
I20260923 20:09:25.385838 6846 ts_tablet_manager.cc:1397] T 41ec739902bf46f887f1ff8bb704120f P ea61b75cc55742aaa8987ef1f487979e: Time spent bootstrapping tablet: real 0.020s user 0.005s sys 0.015s
I20260923 20:09:25.386518 6845 tablet_bootstrap.cc:492] T 41ec739902bf46f887f1ff8bb704120f P e087226ecd1b461c823911514d96bf69: No bootstrap required, opened a new log
I20260923 20:09:25.386828 6845 ts_tablet_manager.cc:1397] T 41ec739902bf46f887f1ff8bb704120f P e087226ecd1b461c823911514d96bf69: Time spent bootstrapping tablet: real 0.022s user 0.003s sys 0.014s
I20260923 20:09:25.394307 6846 raft_consensus.cc:348] T 41ec739902bf46f887f1ff8bb704120f P ea61b75cc55742aaa8987ef1f487979e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e087226ecd1b461c823911514d96bf69" member_type: VOTER last_known_addr { host: "127.1.170.130" port: 36217 } } peers { permanent_uuid: "ea61b75cc55742aaa8987ef1f487979e" member_type: VOTER last_known_addr { host: "127.1.170.129" port: 40663 } } peers { permanent_uuid: "50602caf24974536bb98d52fc5d2ab3f" member_type: VOTER last_known_addr { host: "127.1.170.131" port: 33747 } }
I20260923 20:09:25.394738 6846 raft_consensus.cc:374] T 41ec739902bf46f887f1ff8bb704120f P ea61b75cc55742aaa8987ef1f487979e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:09:25.394865 6846 raft_consensus.cc:729] T 41ec739902bf46f887f1ff8bb704120f P ea61b75cc55742aaa8987ef1f487979e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea61b75cc55742aaa8987ef1f487979e, State: Initialized, Role: FOLLOWER
I20260923 20:09:25.394693 6844 raft_consensus.cc:348] T 41ec739902bf46f887f1ff8bb704120f P 50602caf24974536bb98d52fc5d2ab3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e087226ecd1b461c823911514d96bf69" member_type: VOTER last_known_addr { host: "127.1.170.130" port: 36217 } } peers { permanent_uuid: "ea61b75cc55742aaa8987ef1f487979e" member_type: VOTER last_known_addr { host: "127.1.170.129" port: 40663 } } peers { permanent_uuid: "50602caf24974536bb98d52fc5d2ab3f" member_type: VOTER last_known_addr { host: "127.1.170.131" port: 33747 } }
I20260923 20:09:25.395103 6844 raft_consensus.cc:374] T 41ec739902bf46f887f1ff8bb704120f P 50602caf24974536bb98d52fc5d2ab3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:09:25.395525 6844 raft_consensus.cc:729] T 41ec739902bf46f887f1ff8bb704120f P 50602caf24974536bb98d52fc5d2ab3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 50602caf24974536bb98d52fc5d2ab3f, State: Initialized, Role: FOLLOWER
I20260923 20:09:25.395568 6846 consensus_queue.cc:260] T 41ec739902bf46f887f1ff8bb704120f P ea61b75cc55742aaa8987ef1f487979e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e087226ecd1b461c823911514d96bf69" member_type: VOTER last_known_addr { host: "127.1.170.130" port: 36217 } } peers { permanent_uuid: "ea61b75cc55742aaa8987ef1f487979e" member_type: VOTER last_known_addr { host: "127.1.170.129" port: 40663 } } peers { permanent_uuid: "50602caf24974536bb98d52fc5d2ab3f" member_type: VOTER last_known_addr { host: "127.1.170.131" port: 33747 } }
I20260923 20:09:25.396250 6844 consensus_queue.cc:260] T 41ec739902bf46f887f1ff8bb704120f P 50602caf24974536bb98d52fc5d2ab3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e087226ecd1b461c823911514d96bf69" member_type: VOTER last_known_addr { host: "127.1.170.130" port: 36217 } } peers { permanent_uuid: "ea61b75cc55742aaa8987ef1f487979e" member_type: VOTER last_known_addr { host: "127.1.170.129" port: 40663 } } peers { permanent_uuid: "50602caf24974536bb98d52fc5d2ab3f" member_type: VOTER last_known_addr { host: "127.1.170.131" port: 33747 } }
I20260923 20:09:25.397599 6845 raft_consensus.cc:348] T 41ec739902bf46f887f1ff8bb704120f P e087226ecd1b461c823911514d96bf69 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e087226ecd1b461c823911514d96bf69" member_type: VOTER last_known_addr { host: "127.1.170.130" port: 36217 } } peers { permanent_uuid: "ea61b75cc55742aaa8987ef1f487979e" member_type: VOTER last_known_addr { host: "127.1.170.129" port: 40663 } } peers { permanent_uuid: "50602caf24974536bb98d52fc5d2ab3f" member_type: VOTER last_known_addr { host: "127.1.170.131" port: 33747 } }
I20260923 20:09:25.397980 6826 heartbeater.cc:503] Master 127.1.170.190:34991 was elected leader, sending a full tablet report...
I20260923 20:09:25.398128 6845 raft_consensus.cc:374] T 41ec739902bf46f887f1ff8bb704120f P e087226ecd1b461c823911514d96bf69 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:09:25.398222 6845 raft_consensus.cc:729] T 41ec739902bf46f887f1ff8bb704120f P e087226ecd1b461c823911514d96bf69 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e087226ecd1b461c823911514d96bf69, State: Initialized, Role: FOLLOWER
I20260923 20:09:25.398627 6844 ts_tablet_manager.cc:1428] T 41ec739902bf46f887f1ff8bb704120f P 50602caf24974536bb98d52fc5d2ab3f: Time spent starting tablet: real 0.016s user 0.013s sys 0.001s
I20260923 20:09:25.398795 6845 consensus_queue.cc:260] T 41ec739902bf46f887f1ff8bb704120f P e087226ecd1b461c823911514d96bf69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e087226ecd1b461c823911514d96bf69" member_type: VOTER last_known_addr { host: "127.1.170.130" port: 36217 } } peers { permanent_uuid: "ea61b75cc55742aaa8987ef1f487979e" member_type: VOTER last_known_addr { host: "127.1.170.129" port: 40663 } } peers { permanent_uuid: "50602caf24974536bb98d52fc5d2ab3f" member_type: VOTER last_known_addr { host: "127.1.170.131" port: 33747 } }
I20260923 20:09:25.399631 6846 ts_tablet_manager.cc:1428] T 41ec739902bf46f887f1ff8bb704120f P ea61b75cc55742aaa8987ef1f487979e: Time spent starting tablet: real 0.014s user 0.012s sys 0.000s
I20260923 20:09:25.401333 6696 heartbeater.cc:503] Master 127.1.170.190:34991 was elected leader, sending a full tablet report...
I20260923 20:09:25.401955 6845 ts_tablet_manager.cc:1428] T 41ec739902bf46f887f1ff8bb704120f P e087226ecd1b461c823911514d96bf69: Time spent starting tablet: real 0.015s user 0.012s sys 0.003s
W20260923 20:09:25.533612 6827 tablet.cc:2369] T 41ec739902bf46f887f1ff8bb704120f P 50602caf24974536bb98d52fc5d2ab3f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260923 20:09:25.554028 6697 tablet.cc:2369] T 41ec739902bf46f887f1ff8bb704120f P e087226ecd1b461c823911514d96bf69: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260923 20:09:25.619555 6566 tablet.cc:2369] T 41ec739902bf46f887f1ff8bb704120f P ea61b75cc55742aaa8987ef1f487979e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 20:09:25.642349 6851 raft_consensus.cc:482] T 41ec739902bf46f887f1ff8bb704120f P 50602caf24974536bb98d52fc5d2ab3f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 20:09:25.642674 6851 raft_consensus.cc:504] T 41ec739902bf46f887f1ff8bb704120f P 50602caf24974536bb98d52fc5d2ab3f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e087226ecd1b461c823911514d96bf69" member_type: VOTER last_known_addr { host: "127.1.170.130" port: 36217 } } peers { permanent_uuid: "ea61b75cc55742aaa8987ef1f487979e" member_type: VOTER last_known_addr { host: "127.1.170.129" port: 40663 } } peers { permanent_uuid: "50602caf24974536bb98d52fc5d2ab3f" member_type: VOTER last_known_addr { host: "127.1.170.131" port: 33747 } }
I20260923 20:09:25.644378 6851 leader_election.cc:288] T 41ec739902bf46f887f1ff8bb704120f P 50602caf24974536bb98d52fc5d2ab3f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e087226ecd1b461c823911514d96bf69 (127.1.170.130:36217), ea61b75cc55742aaa8987ef1f487979e (127.1.170.129:40663)
I20260923 20:09:25.647668 6838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:09:25.644728 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.170.131:41209 (local address 127.1.170.130:36217)
0923 20:09:25.644940 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0923 20:09:25.644948 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 20:09:25.645072 (+ 124us) server_negotiation.cc:407] Connection header received
0923 20:09:25.645298 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:09:25.645304 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:09:25.645373 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:09:25.645488 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:09:25.646684 (+ 1196us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:09:25.646690 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:09:25.646694 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 20:09:25.646721 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:09:25.646770 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 20:09:25.646786 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:09:25.646950 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0923 20:09:25.647332 (+ 382us) server_negotiation.cc:299] Negotiation successful
0923 20:09:25.647407 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95}
I20260923 20:09:25.647774 6828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:09:25.644687 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.130:36217 (local address 127.1.170.131:41209)
0923 20:09:25.644940 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0923 20:09:25.644989 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0923 20:09:25.645119 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:09:25.645526 (+ 407us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:09:25.645531 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:09:25.645560 (+ 29us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:09:25.646266 (+ 706us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 20:09:25.646271 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:09:25.646295 (+ 24us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 20:09:25.646351 (+ 56us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:09:25.647083 (+ 732us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:09:25.647094 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:09:25.647103 (+ 9us) client_negotiation.cc:769] Sending connection context
0923 20:09:25.647282 (+ 179us) client_negotiation.cc:240] Negotiation successful
0923 20:09:25.647358 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148}
I20260923 20:09:25.648608 6840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:09:25.644907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.170.131:59971 (local address 127.1.170.129:40663)
0923 20:09:25.645103 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0923 20:09:25.645112 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 20:09:25.645635 (+ 523us) server_negotiation.cc:407] Connection header received
0923 20:09:25.645755 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:09:25.645758 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:09:25.645803 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:09:25.645937 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:09:25.647605 (+ 1668us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:09:25.647613 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:09:25.647619 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 20:09:25.647646 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:09:25.647694 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 20:09:25.647713 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:09:25.647866 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0923 20:09:25.648082 (+ 216us) server_negotiation.cc:299] Negotiation successful
0923 20:09:25.648163 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260923 20:09:25.648811 6853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:09:25.644785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.129:40663 (local address 127.1.170.131:59971)
0923 20:09:25.645526 (+ 741us) negotiation.cc:107] Waiting for socket to connect
0923 20:09:25.645541 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0923 20:09:25.645647 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:09:25.646116 (+ 469us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:09:25.646120 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:09:25.646192 (+ 72us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:09:25.647324 (+ 1132us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 20:09:25.647327 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:09:25.647344 (+ 17us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 20:09:25.647389 (+ 45us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:09:25.647945 (+ 556us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:09:25.647951 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:09:25.647957 (+ 6us) client_negotiation.cc:769] Sending connection context
0923 20:09:25.648111 (+ 154us) client_negotiation.cc:240] Negotiation successful
0923 20:09:25.648172 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":599,"thread_start_us":448,"threads_started":1}
I20260923 20:09:25.648877 6651 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41ec739902bf46f887f1ff8bb704120f" candidate_uuid: "50602caf24974536bb98d52fc5d2ab3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e087226ecd1b461c823911514d96bf69" is_pre_election: true
I20260923 20:09:25.649520 6651 raft_consensus.cc:2454] T 41ec739902bf46f887f1ff8bb704120f P e087226ecd1b461c823911514d96bf69 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 50602caf24974536bb98d52fc5d2ab3f in term 0.
I20260923 20:09:25.649930 6520 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41ec739902bf46f887f1ff8bb704120f" candidate_uuid: "50602caf24974536bb98d52fc5d2ab3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea61b75cc55742aaa8987ef1f487979e" is_pre_election: true
I20260923 20:09:25.650314 6717 leader_election.cc:302] T 41ec739902bf46f887f1ff8bb704120f P 50602caf24974536bb98d52fc5d2ab3f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 50602caf24974536bb98d52fc5d2ab3f, e087226ecd1b461c823911514d96bf69; no voters:
I20260923 20:09:25.650482 6520 raft_consensus.cc:2454] T 41ec739902bf46f887f1ff8bb704120f P ea61b75cc55742aaa8987ef1f487979e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 50602caf24974536bb98d52fc5d2ab3f in term 0.
I20260923 20:09:25.650776 6851 raft_consensus.cc:2781] T 41ec739902bf46f887f1ff8bb704120f P 50602caf24974536bb98d52fc5d2ab3f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 20:09:25.650933 6851 raft_consensus.cc:482] T 41ec739902bf46f887f1ff8bb704120f P 50602caf24974536bb98d52fc5d2ab3f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 20:09:25.651043 6851 raft_consensus.cc:3037] T 41ec739902bf46f887f1ff8bb704120f P 50602caf24974536bb98d52fc5d2ab3f [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:09:25.653290 6851 raft_consensus.cc:504] T 41ec739902bf46f887f1ff8bb704120f P 50602caf24974536bb98d52fc5d2ab3f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e087226ecd1b461c823911514d96bf69" member_type: VOTER last_known_addr { host: "127.1.170.130" port: 36217 } } peers { permanent_uuid: "ea61b75cc55742aaa8987ef1f487979e" member_type: VOTER last_known_addr { host: "127.1.170.129" port: 40663 } } peers { permanent_uuid: "50602caf24974536bb98d52fc5d2ab3f" member_type: VOTER last_known_addr { host: "127.1.170.131" port: 33747 } }
I20260923 20:09:25.654002 6851 leader_election.cc:288] T 41ec739902bf46f887f1ff8bb704120f P 50602caf24974536bb98d52fc5d2ab3f [CANDIDATE]: Term 1 election: Requested vote from peers e087226ecd1b461c823911514d96bf69 (127.1.170.130:36217), ea61b75cc55742aaa8987ef1f487979e (127.1.170.129:40663)
I20260923 20:09:25.654561 6520 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41ec739902bf46f887f1ff8bb704120f" candidate_uuid: "50602caf24974536bb98d52fc5d2ab3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea61b75cc55742aaa8987ef1f487979e"
I20260923 20:09:25.654561 6651 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41ec739902bf46f887f1ff8bb704120f" candidate_uuid: "50602caf24974536bb98d52fc5d2ab3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e087226ecd1b461c823911514d96bf69"
I20260923 20:09:25.654793 6520 raft_consensus.cc:3037] T 41ec739902bf46f887f1ff8bb704120f P ea61b75cc55742aaa8987ef1f487979e [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:09:25.654779 6651 raft_consensus.cc:3037] T 41ec739902bf46f887f1ff8bb704120f P e087226ecd1b461c823911514d96bf69 [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:09:25.658025 6520 raft_consensus.cc:2454] T 41ec739902bf46f887f1ff8bb704120f P ea61b75cc55742aaa8987ef1f487979e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 50602caf24974536bb98d52fc5d2ab3f in term 1.
I20260923 20:09:25.658025 6651 raft_consensus.cc:2454] T 41ec739902bf46f887f1ff8bb704120f P e087226ecd1b461c823911514d96bf69 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 50602caf24974536bb98d52fc5d2ab3f in term 1.
I20260923 20:09:25.658547 6715 leader_election.cc:302] T 41ec739902bf46f887f1ff8bb704120f P 50602caf24974536bb98d52fc5d2ab3f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 50602caf24974536bb98d52fc5d2ab3f, ea61b75cc55742aaa8987ef1f487979e; no voters:
I20260923 20:09:25.658913 6851 raft_consensus.cc:2781] T 41ec739902bf46f887f1ff8bb704120f P 50602caf24974536bb98d52fc5d2ab3f [term 1 FOLLOWER]: Leader election won for term 1
I20260923 20:09:25.659385 6851 raft_consensus.cc:686] T 41ec739902bf46f887f1ff8bb704120f P 50602caf24974536bb98d52fc5d2ab3f [term 1 LEADER]: Becoming Leader. State: Replica: 50602caf24974536bb98d52fc5d2ab3f, State: Running, Role: LEADER
I20260923 20:09:25.659900 6851 consensus_queue.cc:237] T 41ec739902bf46f887f1ff8bb704120f P 50602caf24974536bb98d52fc5d2ab3f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e087226ecd1b461c823911514d96bf69" member_type: VOTER last_known_addr { host: "127.1.170.130" port: 36217 } } peers { permanent_uuid: "ea61b75cc55742aaa8987ef1f487979e" member_type: VOTER last_known_addr { host: "127.1.170.129" port: 40663 } } peers { permanent_uuid: "50602caf24974536bb98d52fc5d2ab3f" member_type: VOTER last_known_addr { host: "127.1.170.131" port: 33747 } }
I20260923 20:09:25.665400 6379 catalog_manager.cc:5495] T 41ec739902bf46f887f1ff8bb704120f P 50602caf24974536bb98d52fc5d2ab3f reported cstate change: term changed from 0 to 1, leader changed from <none> to 50602caf24974536bb98d52fc5d2ab3f (127.1.170.131). New cstate: current_term: 1 leader_uuid: "50602caf24974536bb98d52fc5d2ab3f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e087226ecd1b461c823911514d96bf69" member_type: VOTER last_known_addr { host: "127.1.170.130" port: 36217 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ea61b75cc55742aaa8987ef1f487979e" member_type: VOTER last_known_addr { host: "127.1.170.129" port: 40663 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "50602caf24974536bb98d52fc5d2ab3f" member_type: VOTER last_known_addr { host: "127.1.170.131" port: 33747 } health_report { overall_health: HEALTHY } } }
I20260923 20:09:25.724165 6843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:09:25.721583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45206 (local address 127.1.170.131:33747)
0923 20:09:25.721775 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0923 20:09:25.721801 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0923 20:09:25.721820 (+ 19us) server_negotiation.cc:407] Connection header received
0923 20:09:25.721937 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:09:25.721940 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:09:25.722043 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:09:25.722130 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:09:25.723429 (+ 1299us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:09:25.723433 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:09:25.723437 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 20:09:25.723455 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:09:25.723492 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 20:09:25.723505 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:09:25.723676 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0923 20:09:25.723797 (+ 121us) server_negotiation.cc:299] Negotiation successful
0923 20:09:25.723849 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260923 20:09:25.733806 6651 raft_consensus.cc:1261] T 41ec739902bf46f887f1ff8bb704120f P e087226ecd1b461c823911514d96bf69 [term 1 FOLLOWER]: Refusing update from remote peer 50602caf24974536bb98d52fc5d2ab3f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 20:09:25.734707 6520 raft_consensus.cc:1261] T 41ec739902bf46f887f1ff8bb704120f P ea61b75cc55742aaa8987ef1f487979e [term 1 FOLLOWER]: Refusing update from remote peer 50602caf24974536bb98d52fc5d2ab3f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 20:09:25.735169 6854 consensus_queue.cc:1035] T 41ec739902bf46f887f1ff8bb704120f P 50602caf24974536bb98d52fc5d2ab3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "e087226ecd1b461c823911514d96bf69" member_type: VOTER last_known_addr { host: "127.1.170.130" port: 36217 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 20:09:25.735643 6851 consensus_queue.cc:1035] T 41ec739902bf46f887f1ff8bb704120f P 50602caf24974536bb98d52fc5d2ab3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea61b75cc55742aaa8987ef1f487979e" member_type: VOTER last_known_addr { host: "127.1.170.129" port: 40663 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 20:09:25.748447 6857 mvcc.cc:205] Tried to move back new op lower bound from 7332635302832594944 to 7332635302547783680. Current Snapshot: MvccSnapshot[applied={T|T < 7332635302832594944}]
I20260923 20:09:25.750195 6859 mvcc.cc:205] Tried to move back new op lower bound from 7332635302832594944 to 7332635302547783680. Current Snapshot: MvccSnapshot[applied={T|T < 7332635302832594944}]
I20260923 20:09:25.765743 6379 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44892:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 20:09:25.766171 6379 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44892:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 20:09:25.771075 6379 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f3fc7212a9c1456da161d7c108ad1f75: Sending DeleteTablet for 3 replicas of tablet 41ec739902bf46f887f1ff8bb704120f
I20260923 20:09:25.772358 6500 tablet_service.cc:1514] Processing DeleteTablet for tablet 41ec739902bf46f887f1ff8bb704120f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 20:09:25 UTC) from {username='slave'} at 127.0.0.1:46324
I20260923 20:09:25.772362 6631 tablet_service.cc:1514] Processing DeleteTablet for tablet 41ec739902bf46f887f1ff8bb704120f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 20:09:25 UTC) from {username='slave'} at 127.0.0.1:32940
I20260923 20:09:25.773133 6761 tablet_service.cc:1514] Processing DeleteTablet for tablet 41ec739902bf46f887f1ff8bb704120f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 20:09:25 UTC) from {username='slave'} at 127.0.0.1:45190
I20260923 20:09:25.774149 6869 tablet_replica.cc:330] T 41ec739902bf46f887f1ff8bb704120f P 50602caf24974536bb98d52fc5d2ab3f: stopping tablet replica
I20260923 20:09:25.774773 6869 raft_consensus.cc:2229] T 41ec739902bf46f887f1ff8bb704120f P 50602caf24974536bb98d52fc5d2ab3f [term 1 LEADER]: Raft consensus shutting down.
I20260923 20:09:25.775756 6869 raft_consensus.cc:2258] T 41ec739902bf46f887f1ff8bb704120f P 50602caf24974536bb98d52fc5d2ab3f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 20:09:25.778216 6869 ts_tablet_manager.cc:1905] T 41ec739902bf46f887f1ff8bb704120f P 50602caf24974536bb98d52fc5d2ab3f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 20:09:25.778390 1706 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskqUuku6/build/asan/bin/kudu pid 6438
I20260923 20:09:25.789179 6840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:09:25.780017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46332 (local address 127.1.170.129:40663)
0923 20:09:25.780265 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0923 20:09:25.780274 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 20:09:25.784704 (+ 4430us) server_negotiation.cc:407] Connection header received
0923 20:09:25.785830 (+ 1126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:09:25.785838 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:09:25.785986 (+ 148us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:09:25.786066 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:09:25.788117 (+ 2051us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:09:25.788123 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:09:25.788128 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 20:09:25.788149 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:09:25.788193 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 20:09:25.788214 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:09:25.788324 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0923 20:09:25.788740 (+ 416us) server_negotiation.cc:299] Negotiation successful
0923 20:09:25.788822 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260923 20:09:25.789419 6869 ts_tablet_manager.cc:1918] T 41ec739902bf46f887f1ff8bb704120f P 50602caf24974536bb98d52fc5d2ab3f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 20:09:25.789716 6869 log.cc:1196] T 41ec739902bf46f887f1ff8bb704120f P 50602caf24974536bb98d52fc5d2ab3f: Deleting WAL directory at /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790194121577027-1706-0/minicluster-data/ts-2/wal/wals/41ec739902bf46f887f1ff8bb704120f
I20260923 20:09:25.790442 6869 ts_tablet_manager.cc:1939] T 41ec739902bf46f887f1ff8bb704120f P 50602caf24974536bb98d52fc5d2ab3f: Deleting consensus metadata
I20260923 20:09:25.790560 6560 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:09:25.790889 6868 tablet_replica.cc:330] T 41ec739902bf46f887f1ff8bb704120f P e087226ecd1b461c823911514d96bf69: stopping tablet replica
I20260923 20:09:25.792104 6868 raft_consensus.cc:2229] T 41ec739902bf46f887f1ff8bb704120f P e087226ecd1b461c823911514d96bf69 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 20:09:25.792991 6364 catalog_manager.cc:4841] TS 50602caf24974536bb98d52fc5d2ab3f (127.1.170.131:33747): tablet 41ec739902bf46f887f1ff8bb704120f (table test-table [id=fec47bb63d194e26ba128e3d8c421505]) successfully deleted
I20260923 20:09:25.793180 6868 raft_consensus.cc:2258] T 41ec739902bf46f887f1ff8bb704120f P e087226ecd1b461c823911514d96bf69 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 20:09:25.796468 6868 ts_tablet_manager.cc:1905] T 41ec739902bf46f887f1ff8bb704120f P e087226ecd1b461c823911514d96bf69: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 20:09:25.819921 6868 ts_tablet_manager.cc:1918] T 41ec739902bf46f887f1ff8bb704120f P e087226ecd1b461c823911514d96bf69: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 20:09:25.820226 6868 log.cc:1196] T 41ec739902bf46f887f1ff8bb704120f P e087226ecd1b461c823911514d96bf69: Deleting WAL directory at /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790194121577027-1706-0/minicluster-data/ts-1/wal/wals/41ec739902bf46f887f1ff8bb704120f
I20260923 20:09:25.820876 6868 ts_tablet_manager.cc:1939] T 41ec739902bf46f887f1ff8bb704120f P e087226ecd1b461c823911514d96bf69: Deleting consensus metadata
I20260923 20:09:25.823073 6366 catalog_manager.cc:4841] TS e087226ecd1b461c823911514d96bf69 (127.1.170.130:36217): tablet 41ec739902bf46f887f1ff8bb704120f (table test-table [id=fec47bb63d194e26ba128e3d8c421505]) successfully deleted
I20260923 20:09:25.919657 6867 tablet_replica.cc:330] T 41ec739902bf46f887f1ff8bb704120f P ea61b75cc55742aaa8987ef1f487979e: stopping tablet replica
I20260923 20:09:25.919867 1706 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqUuku6/build/asan/bin/kudu with pid 6438
W20260923 20:09:25.938851 6364 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.1.170.129:40663 (error 108)
I20260923 20:09:25.939850 1706 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskqUuku6/build/asan/bin/kudu pid 6569
I20260923 20:09:25.941008 6873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:09:25.940132 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.129:40663 (local address 127.0.0.1:46340)
0923 20:09:25.940589 (+ 457us) negotiation.cc:107] Waiting for socket to connect
0923 20:09:25.940820 (+ 231us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.170.129:40663: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":322,"thread_start_us":239,"threads_started":1}
W20260923 20:09:25.941465 6364 catalog_manager.cc:4584] TS ea61b75cc55742aaa8987ef1f487979e (127.1.170.129:40663): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 41ec739902bf46f887f1ff8bb704120f: Network error: Client connection negotiation failed: client connection to 127.1.170.129:40663: connect: Connection refused (error 111)
I20260923 20:09:25.943758 6838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:09:25.941199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32952 (local address 127.1.170.130:36217)
0923 20:09:25.941381 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0923 20:09:25.941386 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 20:09:25.941406 (+ 20us) server_negotiation.cc:407] Connection header received
0923 20:09:25.941466 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:09:25.941470 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:09:25.941568 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:09:25.941710 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:09:25.942728 (+ 1018us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:09:25.942734 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:09:25.942740 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 20:09:25.942764 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:09:25.942806 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 20:09:25.942822 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:09:25.942953 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0923 20:09:25.943389 (+ 436us) server_negotiation.cc:299] Negotiation successful
0923 20:09:25.943452 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260923 20:09:25.944730 6691 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:09:25.995693 6873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:09:25.994951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.129:40663 (local address 127.0.0.1:46354)
0923 20:09:25.995408 (+ 457us) negotiation.cc:107] Waiting for socket to connect
0923 20:09:25.995513 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.170.129:40663: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":257}
I20260923 20:09:25.997160 6875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:09:25.996348 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.129:40663 (local address 127.0.0.1:46362)
0923 20:09:25.996826 (+ 478us) negotiation.cc:107] Waiting for socket to connect
0923 20:09:25.996943 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.170.129:40663: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":355,"thread_start_us":234,"threads_started":1}
I20260923 20:09:26.055155 1706 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqUuku6/build/asan/bin/kudu with pid 6569
I20260923 20:09:26.071043 6875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:09:26.070470 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.129:40663 (local address 127.0.0.1:46374)
0923 20:09:26.070710 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0923 20:09:26.070862 (+ 152us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.170.129:40663: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
I20260923 20:09:26.072371 6875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:09:26.071859 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.129:40663 (local address 127.0.0.1:46380)
0923 20:09:26.072082 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0923 20:09:26.072185 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.170.129:40663: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260923 20:09:26.073366 1706 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskqUuku6/build/asan/bin/kudu pid 6699
I20260923 20:09:26.076221 6843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:09:26.074112 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45212 (local address 127.1.170.131:33747)
0923 20:09:26.074278 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0923 20:09:26.074286 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 20:09:26.074305 (+ 19us) server_negotiation.cc:407] Connection header received
0923 20:09:26.074564 (+ 259us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:09:26.074568 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:09:26.074618 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:09:26.074741 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:09:26.075592 (+ 851us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:09:26.075596 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:09:26.075599 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 20:09:26.075613 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:09:26.075640 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 20:09:26.075650 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:09:26.075756 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0923 20:09:26.075869 (+ 113us) server_negotiation.cc:299] Negotiation successful
0923 20:09:26.075910 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260923 20:09:26.076936 6821 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:09:26.153977 6875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:09:26.153357 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.129:40663 (local address 127.0.0.1:46388)
0923 20:09:26.153618 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0923 20:09:26.153775 (+ 157us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.170.129:40663: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95}
I20260923 20:09:26.155153 6875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:09:26.154676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.129:40663 (local address 127.0.0.1:46398)
0923 20:09:26.154901 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0923 20:09:26.154983 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.170.129:40663: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109}
I20260923 20:09:26.200850 1706 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqUuku6/build/asan/bin/kudu with pid 6699
I20260923 20:09:26.219643 1706 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskqUuku6/build/asan/bin/kudu pid 6348
I20260923 20:09:26.221005 6409 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:09:26.327382 1706 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqUuku6/build/asan/bin/kudu with pid 6348
2026-09-23T20:09:26Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (2952 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260923 20:09:26.364678 1706 mini_oidc.cc:152] Starting JWKS server
I20260923 20:09:26.367401 1706 webserver.cc:460] Webserver started at https://127.0.0.1:36127/ using document root <none> and password file <none>
I20260923 20:09:26.367710 1706 mini_oidc.cc:189] Starting OIDC Discovery server
I20260923 20:09:26.368400 1706 webserver.cc:460] Webserver started at http://127.0.0.1:35965/ using document root <none> and password file <none>
2026-09-23T20:09:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T20:09:26Z Disabled control of system clock
I20260923 20:09:26.394564 1706 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqUuku6/build/asan/bin/kudu
/tmp/dist-test-taskqUuku6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790194121577027-1706-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790194121577027-1706-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790194121577027-1706-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790194121577027-1706-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.170.190:37865
--webserver_interface=127.1.170.190
--webserver_port=0
--builtin_ntp_servers=127.1.170.148:46691
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.170.190:37865
--enable_jwt_token_auth=true
--jwks_url=https://localhost:36127/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790194121577027-1706-0/testchainca.cert with env {}
W20260923 20:09:26.716084 6883 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:09:26.716377 6883 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:09:26.716527 6883 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:09:26.723796 6883 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 20:09:26.723907 6883 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:09:26.723987 6883 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:09:26.724080 6883 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 20:09:26.724155 6883 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260923 20:09:26.724251 6883 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260923 20:09:26.724303 6883 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:36127/jwks.json
I20260923 20:09:26.734412 6883 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.170.148:46691
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790194121577027-1706-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790194121577027-1706-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.170.190:37865
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.170.190:37865
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:36127/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790194121577027-1706-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.170.190
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790194121577027-1706-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790194121577027-1706-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:09:26.735734 6883 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:09:26.737074 6883 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:09:26.748519 6888 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:09:26.748464 6891 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:09:26.748498 6889 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:09:26.749526 6883 server_base.cc:1034] running on GCE node
I20260923 20:09:26.750352 6883 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:09:26.751717 6883 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:09:26.752928 6883 hybrid_clock.cc:648] HybridClock initialized: now 1790194166752890 us; error 31 us; skew 500 ppm
I20260923 20:09:26.753221 6883 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260923 20:09:26.755462 6883 webserver.cc:460] Webserver started at http://127.1.170.190:41979/ using document root <none> and password file <none>
I20260923 20:09:26.756080 6883 fs_manager.cc:359] Metadata directory not provided
I20260923 20:09:26.756168 6883 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:09:26.756419 6883 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:09:26.759258 6883 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790194121577027-1706-0/minicluster-data/master-0/data/instance:
uuid: "c376f65ba5974073bd7a6d8bb88179e4"
format_stamp: "Formatted at 2026-09-23 20:09:26 on dist-test-slave-th4g"
server_key: "4784a0cbe391559494fc04d36e0f9ae9"
server_key_iv: "66969788c46edf410771912efd95bfc8"
server_key_version: "encryptionkey@0"
I20260923 20:09:26.759992 6883 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790194121577027-1706-0/minicluster-data/master-0/wal/instance:
uuid: "c376f65ba5974073bd7a6d8bb88179e4"
format_stamp: "Formatted at 2026-09-23 20:09:26 on dist-test-slave-th4g"
server_key: "4784a0cbe391559494fc04d36e0f9ae9"
server_key_iv: "66969788c46edf410771912efd95bfc8"
server_key_version: "encryptionkey@0"
I20260923 20:09:26.764779 6883 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260923 20:09:26.768070 6897 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:09:26.769347 6883 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260923 20:09:26.769511 6883 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790194121577027-1706-0/minicluster-data/master-0/data,/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790194121577027-1706-0/minicluster-data/master-0/wal
uuid: "c376f65ba5974073bd7a6d8bb88179e4"
format_stamp: "Formatted at 2026-09-23 20:09:26 on dist-test-slave-th4g"
server_key: "4784a0cbe391559494fc04d36e0f9ae9"
server_key_iv: "66969788c46edf410771912efd95bfc8"
server_key_version: "encryptionkey@0"
I20260923 20:09:26.769714 6883 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790194121577027-1706-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790194121577027-1706-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790194121577027-1706-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:09:26.783288 6883 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:09:26.784092 6883 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:09:26.784401 6883 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:09:26.802702 6883 rpc_server.cc:307] RPC server started. Bound to: 127.1.170.190:37865
I20260923 20:09:26.802723 6948 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.170.190:37865 every 8 connection(s)
I20260923 20:09:26.804103 6883 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790194121577027-1706-0/minicluster-data/master-0/data/info.pb
I20260923 20:09:26.807917 6949 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:09:26.812945 1706 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqUuku6/build/asan/bin/kudu as pid 6883
I20260923 20:09:26.813160 1706 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790194121577027-1706-0/minicluster-data/master-0/wal/instance
I20260923 20:09:26.813788 1706 external_mini_cluster.cc:1428] Setting key 6dae8ae1c9bb7fbebed62ef94425b0c3
I20260923 20:09:26.818895 6949 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c376f65ba5974073bd7a6d8bb88179e4: Bootstrap starting.
I20260923 20:09:26.822476 6949 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c376f65ba5974073bd7a6d8bb88179e4: Neither blocks nor log segments found. Creating new log.
I20260923 20:09:26.823676 6949 log.cc:824] T 00000000000000000000000000000000 P c376f65ba5974073bd7a6d8bb88179e4: Log is configured to *not* fsync() on all Append() calls
I20260923 20:09:26.826943 6949 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c376f65ba5974073bd7a6d8bb88179e4: No bootstrap required, opened a new log
I20260923 20:09:26.827920 6952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:09:26.815899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44452 (local address 127.1.170.190:37865)
0923 20:09:26.816920 (+ 1021us) server_negotiation.cc:206] Beginning negotiation
0923 20:09:26.816931 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0923 20:09:26.816975 (+ 44us) server_negotiation.cc:407] Connection header received
0923 20:09:26.818678 (+ 1703us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:09:26.818699 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:09:26.819175 (+ 476us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:09:26.819487 (+ 312us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:09:26.820163 (+ 676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:09:26.821473 (+ 1310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:09:26.823351 (+ 1878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:09:26.824085 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:09:26.824303 (+ 218us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:09:26.825521 (+ 1218us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:09:26.825529 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:09:26.825542 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 20:09:26.826140 (+ 598us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:09:26.826211 (+ 71us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 20:09:26.826236 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:09:26.826329 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0923 20:09:26.826596 (+ 267us) server_negotiation.cc:299] Negotiation successful
0923 20:09:26.826921 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":602,"thread_start_us":388,"threads_started":1}
I20260923 20:09:26.831507 1706 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 20:09:26.835067 6949 raft_consensus.cc:348] T 00000000000000000000000000000000 P c376f65ba5974073bd7a6d8bb88179e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c376f65ba5974073bd7a6d8bb88179e4" member_type: VOTER last_known_addr { host: "127.1.170.190" port: 37865 } }
I20260923 20:09:26.835434 6949 raft_consensus.cc:374] T 00000000000000000000000000000000 P c376f65ba5974073bd7a6d8bb88179e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:09:26.835551 6949 raft_consensus.cc:729] T 00000000000000000000000000000000 P c376f65ba5974073bd7a6d8bb88179e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c376f65ba5974073bd7a6d8bb88179e4, State: Initialized, Role: FOLLOWER
I20260923 20:09:26.836124 6949 consensus_queue.cc:260] T 00000000000000000000000000000000 P c376f65ba5974073bd7a6d8bb88179e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c376f65ba5974073bd7a6d8bb88179e4" member_type: VOTER last_known_addr { host: "127.1.170.190" port: 37865 } }
I20260923 20:09:26.836314 6949 raft_consensus.cc:388] T 00000000000000000000000000000000 P c376f65ba5974073bd7a6d8bb88179e4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 20:09:26.836479 6949 raft_consensus.cc:482] T 00000000000000000000000000000000 P c376f65ba5974073bd7a6d8bb88179e4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 20:09:26.836638 6949 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c376f65ba5974073bd7a6d8bb88179e4 [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:09:26.838864 6949 raft_consensus.cc:504] T 00000000000000000000000000000000 P c376f65ba5974073bd7a6d8bb88179e4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c376f65ba5974073bd7a6d8bb88179e4" member_type: VOTER last_known_addr { host: "127.1.170.190" port: 37865 } }
I20260923 20:09:26.839411 6949 leader_election.cc:302] T 00000000000000000000000000000000 P c376f65ba5974073bd7a6d8bb88179e4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c376f65ba5974073bd7a6d8bb88179e4; no voters:
I20260923 20:09:26.839980 6949 leader_election.cc:288] T 00000000000000000000000000000000 P c376f65ba5974073bd7a6d8bb88179e4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 20:09:26.840131 6954 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c376f65ba5974073bd7a6d8bb88179e4 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 20:09:26.841552 6954 raft_consensus.cc:686] T 00000000000000000000000000000000 P c376f65ba5974073bd7a6d8bb88179e4 [term 1 LEADER]: Becoming Leader. State: Replica: c376f65ba5974073bd7a6d8bb88179e4, State: Running, Role: LEADER
I20260923 20:09:26.842083 6954 consensus_queue.cc:237] T 00000000000000000000000000000000 P c376f65ba5974073bd7a6d8bb88179e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c376f65ba5974073bd7a6d8bb88179e4" member_type: VOTER last_known_addr { host: "127.1.170.190" port: 37865 } }
I20260923 20:09:26.842350 6949 sys_catalog.cc:564] T 00000000000000000000000000000000 P c376f65ba5974073bd7a6d8bb88179e4 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 20:09:26.844686 6956 sys_catalog.cc:455] T 00000000000000000000000000000000 P c376f65ba5974073bd7a6d8bb88179e4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c376f65ba5974073bd7a6d8bb88179e4. Latest consensus state: current_term: 1 leader_uuid: "c376f65ba5974073bd7a6d8bb88179e4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c376f65ba5974073bd7a6d8bb88179e4" member_type: VOTER last_known_addr { host: "127.1.170.190" port: 37865 } } }
I20260923 20:09:26.845654 6956 sys_catalog.cc:458] T 00000000000000000000000000000000 P c376f65ba5974073bd7a6d8bb88179e4 [sys.catalog]: This master's current role is: LEADER
I20260923 20:09:26.845523 6955 sys_catalog.cc:455] T 00000000000000000000000000000000 P c376f65ba5974073bd7a6d8bb88179e4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c376f65ba5974073bd7a6d8bb88179e4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c376f65ba5974073bd7a6d8bb88179e4" member_type: VOTER last_known_addr { host: "127.1.170.190" port: 37865 } } }
I20260923 20:09:26.845896 6955 sys_catalog.cc:458] T 00000000000000000000000000000000 P c376f65ba5974073bd7a6d8bb88179e4 [sys.catalog]: This master's current role is: LEADER
I20260923 20:09:26.847208 6963 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 20:09:26.852705 6963 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 20:09:26.862378 6963 catalog_manager.cc:1347] Generated new cluster ID: eb81fc7b5ec64b7e8c1d6fc2c6b465d2
I20260923 20:09:26.862495 6963 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 20:09:26.904249 6963 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 20:09:26.906071 6963 catalog_manager.cc:1504] Loading token signing keys...
I20260923 20:09:26.919502 6963 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c376f65ba5974073bd7a6d8bb88179e4: Generated new TSK 0
I20260923 20:09:26.920445 6963 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260923 20:09:26.969027 6979 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260923 20:09:26.969339 6979 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260923 20:09:26.970398 6952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:09:26.947387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44460 (local address 127.1.170.190:37865)
0923 20:09:26.947595 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0923 20:09:26.947602 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:09:26.948276 (+ 674us) server_negotiation.cc:407] Connection header received
0923 20:09:26.948564 (+ 288us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:09:26.948570 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:09:26.948669 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:09:26.948868 (+ 199us) server_negotiation.cc:226] Negotiated authn=JWT
0923 20:09:26.949421 (+ 553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:09:26.950628 (+ 1207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:09:26.952094 (+ 1466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:09:26.952859 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:09:26.953002 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:09:26.953605 (+ 603us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0923 20:09:26.969665 (+ 16060us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36127/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0923 20:09:26.970063 (+ 398us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:44460: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36127/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":62}
W20260923 20:09:26.970515 6952 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:44460: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36127/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260923 20:09:26.971782 6978 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 20:09:26.947228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.190:37865 (local address 127.0.0.1:44460)
0923 20:09:26.948079 (+ 851us) negotiation.cc:107] Waiting for socket to connect
0923 20:09:26.948099 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0923 20:09:26.948273 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:09:26.948940 (+ 667us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:09:26.948944 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:09:26.948954 (+ 10us) client_negotiation.cc:189] Negotiated authn=JWT
0923 20:09:26.949260 (+ 306us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:09:26.949268 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:09:26.950796 (+ 1528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:09:26.950800 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:09:26.951941 (+ 1141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:09:26.951952 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:09:26.953063 (+ 1111us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:09:26.953068 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:09:26.953212 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:09:26.953306 (+ 94us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0923 20:09:26.969980 (+ 16674us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36127/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0923 20:09:26.970260 (+ 280us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.1.170.190:37865: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36127/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":673,"thread_start_us":448,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.1.170.190:37865: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36127/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260923 20:09:26.973834 1706 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskqUuku6/build/asan/bin/kudu pid 6883
I20260923 20:09:26.974975 6944 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:09:27.072182 1706 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqUuku6/build/asan/bin/kudu with pid 6883
2026-09-23T20:09:27Z chronyd exiting
I20260923 20:09:27.123047 1706 test_util.cc:175] -----------------------------------------------
I20260923 20:09:27.123169 1706 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790194121577027-1706-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (762 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260923 20:09:27.126931 1706 mini_oidc.cc:152] Starting JWKS server
I20260923 20:09:27.128751 1706 webserver.cc:460] Webserver started at https://127.0.0.1:41461/ using document root <none> and password file <none>
I20260923 20:09:27.129002 1706 mini_oidc.cc:189] Starting OIDC Discovery server
I20260923 20:09:27.129638 1706 webserver.cc:460] Webserver started at http://127.0.0.1:33635/ using document root <none> and password file <none>
2026-09-23T20:09:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T20:09:27Z Disabled control of system clock
I20260923 20:09:27.156718 1706 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqUuku6/build/asan/bin/kudu
/tmp/dist-test-taskqUuku6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790194121577027-1706-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790194121577027-1706-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790194121577027-1706-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790194121577027-1706-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.170.190:42807
--webserver_interface=127.1.170.190
--webserver_port=0
--builtin_ntp_servers=127.1.170.148:38047
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.170.190:42807
--enable_jwt_token_auth=true
--jwks_url=https://localhost:41461/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790194121577027-1706-0/testchainca.cert with env {}
W20260923 20:09:27.479072 6986 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:09:27.479434 6986 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:09:27.479592 6986 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:09:27.486936 6986 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 20:09:27.487097 6986 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:09:27.487192 6986 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:09:27.487304 6986 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 20:09:27.487388 6986 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260923 20:09:27.487480 6986 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260923 20:09:27.487558 6986 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:41461/jwks.json
I20260923 20:09:27.497500 6986 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.170.148:38047
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790194121577027-1706-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790194121577027-1706-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.170.190:42807
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.170.190:42807
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:41461/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790194121577027-1706-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.170.190
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790194121577027-1706-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790194121577027-1706-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:09:27.498819 6986 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:09:27.500087 6986 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:09:27.511806 6992 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:09:27.511919 6991 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:09:27.511895 6994 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:09:27.512709 6986 server_base.cc:1034] running on GCE node
I20260923 20:09:27.513453 6986 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:09:27.514904 6986 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:09:27.516131 6986 hybrid_clock.cc:648] HybridClock initialized: now 1790194167516098 us; error 31 us; skew 500 ppm
I20260923 20:09:27.516525 6986 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260923 20:09:27.518916 6986 webserver.cc:460] Webserver started at http://127.1.170.190:44181/ using document root <none> and password file <none>
I20260923 20:09:27.519598 6986 fs_manager.cc:359] Metadata directory not provided
I20260923 20:09:27.519718 6986 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:09:27.520052 6986 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:09:27.523020 6986 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790194121577027-1706-0/minicluster-data/master-0/data/instance:
uuid: "ae13674ac00e4ec3b349450f1a345758"
format_stamp: "Formatted at 2026-09-23 20:09:27 on dist-test-slave-th4g"
server_key: "e82f4b237b05d4275fb4d3d4f9179bff"
server_key_iv: "26d5229e0f49335e7459a31e38e7c9bd"
server_key_version: "encryptionkey@0"
I20260923 20:09:27.523809 6986 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790194121577027-1706-0/minicluster-data/master-0/wal/instance:
uuid: "ae13674ac00e4ec3b349450f1a345758"
format_stamp: "Formatted at 2026-09-23 20:09:27 on dist-test-slave-th4g"
server_key: "e82f4b237b05d4275fb4d3d4f9179bff"
server_key_iv: "26d5229e0f49335e7459a31e38e7c9bd"
server_key_version: "encryptionkey@0"
I20260923 20:09:27.528932 6986 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260923 20:09:27.532632 7000 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:09:27.534288 6986 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260923 20:09:27.534469 6986 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790194121577027-1706-0/minicluster-data/master-0/data,/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790194121577027-1706-0/minicluster-data/master-0/wal
uuid: "ae13674ac00e4ec3b349450f1a345758"
format_stamp: "Formatted at 2026-09-23 20:09:27 on dist-test-slave-th4g"
server_key: "e82f4b237b05d4275fb4d3d4f9179bff"
server_key_iv: "26d5229e0f49335e7459a31e38e7c9bd"
server_key_version: "encryptionkey@0"
I20260923 20:09:27.534690 6986 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790194121577027-1706-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790194121577027-1706-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790194121577027-1706-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:09:27.588681 6986 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:09:27.589481 6986 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:09:27.589748 6986 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:09:27.608573 6986 rpc_server.cc:307] RPC server started. Bound to: 127.1.170.190:42807
I20260923 20:09:27.608596 7051 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.170.190:42807 every 8 connection(s)
I20260923 20:09:27.609932 6986 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790194121577027-1706-0/minicluster-data/master-0/data/info.pb
I20260923 20:09:27.613579 7052 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:09:27.616236 1706 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqUuku6/build/asan/bin/kudu as pid 6986
I20260923 20:09:27.616459 1706 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790194121577027-1706-0/minicluster-data/master-0/wal/instance
I20260923 20:09:27.617100 1706 external_mini_cluster.cc:1428] Setting key c2056109512ffe0d759ef9fed33db1d5
I20260923 20:09:27.624933 7052 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ae13674ac00e4ec3b349450f1a345758: Bootstrap starting.
I20260923 20:09:27.628144 7052 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ae13674ac00e4ec3b349450f1a345758: Neither blocks nor log segments found. Creating new log.
I20260923 20:09:27.629242 7052 log.cc:824] T 00000000000000000000000000000000 P ae13674ac00e4ec3b349450f1a345758: Log is configured to *not* fsync() on all Append() calls
I20260923 20:09:27.630245 7055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:09:27.619037 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33920 (local address 127.1.170.190:42807)
0923 20:09:27.619974 (+ 937us) server_negotiation.cc:206] Beginning negotiation
0923 20:09:27.619984 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0923 20:09:27.620024 (+ 40us) server_negotiation.cc:407] Connection header received
0923 20:09:27.621620 (+ 1596us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:09:27.621644 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:09:27.622280 (+ 636us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:09:27.622681 (+ 401us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:09:27.623228 (+ 547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:09:27.624478 (+ 1250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:09:27.625780 (+ 1302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:09:27.626543 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:09:27.626759 (+ 216us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:09:27.627897 (+ 1138us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:09:27.627905 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:09:27.627918 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 20:09:27.628508 (+ 590us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:09:27.628592 (+ 84us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 20:09:27.628612 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:09:27.628781 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0923 20:09:27.628929 (+ 148us) server_negotiation.cc:299] Negotiation successful
0923 20:09:27.629210 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":540,"thread_start_us":338,"threads_started":1}
I20260923 20:09:27.632139 7052 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ae13674ac00e4ec3b349450f1a345758: No bootstrap required, opened a new log
I20260923 20:09:27.633905 1706 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 20:09:27.640221 7052 raft_consensus.cc:348] T 00000000000000000000000000000000 P ae13674ac00e4ec3b349450f1a345758 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae13674ac00e4ec3b349450f1a345758" member_type: VOTER last_known_addr { host: "127.1.170.190" port: 42807 } }
I20260923 20:09:27.640801 7052 raft_consensus.cc:374] T 00000000000000000000000000000000 P ae13674ac00e4ec3b349450f1a345758 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:09:27.640944 7052 raft_consensus.cc:729] T 00000000000000000000000000000000 P ae13674ac00e4ec3b349450f1a345758 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae13674ac00e4ec3b349450f1a345758, State: Initialized, Role: FOLLOWER
I20260923 20:09:27.641660 7052 consensus_queue.cc:260] T 00000000000000000000000000000000 P ae13674ac00e4ec3b349450f1a345758 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae13674ac00e4ec3b349450f1a345758" member_type: VOTER last_known_addr { host: "127.1.170.190" port: 42807 } }
I20260923 20:09:27.641930 7052 raft_consensus.cc:388] T 00000000000000000000000000000000 P ae13674ac00e4ec3b349450f1a345758 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 20:09:27.642083 7052 raft_consensus.cc:482] T 00000000000000000000000000000000 P ae13674ac00e4ec3b349450f1a345758 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 20:09:27.642263 7052 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ae13674ac00e4ec3b349450f1a345758 [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:09:27.644505 7052 raft_consensus.cc:504] T 00000000000000000000000000000000 P ae13674ac00e4ec3b349450f1a345758 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae13674ac00e4ec3b349450f1a345758" member_type: VOTER last_known_addr { host: "127.1.170.190" port: 42807 } }
I20260923 20:09:27.645092 7052 leader_election.cc:302] T 00000000000000000000000000000000 P ae13674ac00e4ec3b349450f1a345758 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ae13674ac00e4ec3b349450f1a345758; no voters:
I20260923 20:09:27.645782 7052 leader_election.cc:288] T 00000000000000000000000000000000 P ae13674ac00e4ec3b349450f1a345758 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 20:09:27.645917 7057 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ae13674ac00e4ec3b349450f1a345758 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 20:09:27.646590 7057 raft_consensus.cc:686] T 00000000000000000000000000000000 P ae13674ac00e4ec3b349450f1a345758 [term 1 LEADER]: Becoming Leader. State: Replica: ae13674ac00e4ec3b349450f1a345758, State: Running, Role: LEADER
I20260923 20:09:27.647197 7057 consensus_queue.cc:237] T 00000000000000000000000000000000 P ae13674ac00e4ec3b349450f1a345758 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae13674ac00e4ec3b349450f1a345758" member_type: VOTER last_known_addr { host: "127.1.170.190" port: 42807 } }
I20260923 20:09:27.648108 7052 sys_catalog.cc:564] T 00000000000000000000000000000000 P ae13674ac00e4ec3b349450f1a345758 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 20:09:27.650002 7058 sys_catalog.cc:455] T 00000000000000000000000000000000 P ae13674ac00e4ec3b349450f1a345758 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ae13674ac00e4ec3b349450f1a345758" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae13674ac00e4ec3b349450f1a345758" member_type: VOTER last_known_addr { host: "127.1.170.190" port: 42807 } } }
I20260923 20:09:27.650465 7058 sys_catalog.cc:458] T 00000000000000000000000000000000 P ae13674ac00e4ec3b349450f1a345758 [sys.catalog]: This master's current role is: LEADER
I20260923 20:09:27.652742 7059 sys_catalog.cc:455] T 00000000000000000000000000000000 P ae13674ac00e4ec3b349450f1a345758 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ae13674ac00e4ec3b349450f1a345758. Latest consensus state: current_term: 1 leader_uuid: "ae13674ac00e4ec3b349450f1a345758" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae13674ac00e4ec3b349450f1a345758" member_type: VOTER last_known_addr { host: "127.1.170.190" port: 42807 } } }
I20260923 20:09:27.653275 7059 sys_catalog.cc:458] T 00000000000000000000000000000000 P ae13674ac00e4ec3b349450f1a345758 [sys.catalog]: This master's current role is: LEADER
I20260923 20:09:27.654374 7062 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 20:09:27.661345 7062 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 20:09:27.670241 7062 catalog_manager.cc:1347] Generated new cluster ID: be2a93dae2284def825dda794cdb8a1d
I20260923 20:09:27.670388 7062 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 20:09:27.685774 7062 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 20:09:27.687222 7062 catalog_manager.cc:1504] Loading token signing keys...
I20260923 20:09:27.692842 7062 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ae13674ac00e4ec3b349450f1a345758: Generated new TSK 0
I20260923 20:09:27.693643 7062 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260923 20:09:27.732116 7082 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260923 20:09:27.732264 7082 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260923 20:09:27.733556 7055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:09:27.712760 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33936 (local address 127.1.170.190:42807)
0923 20:09:27.712948 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0923 20:09:27.712955 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:09:27.713334 (+ 379us) server_negotiation.cc:407] Connection header received
0923 20:09:27.713540 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:09:27.713546 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:09:27.713655 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:09:27.713823 (+ 168us) server_negotiation.cc:226] Negotiated authn=JWT
0923 20:09:27.714375 (+ 552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:09:27.715486 (+ 1111us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:09:27.716879 (+ 1393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:09:27.717587 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:09:27.717723 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:09:27.718226 (+ 503us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0923 20:09:27.732838 (+ 14612us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41461/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0923 20:09:27.733261 (+ 423us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:33936: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41461/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":68}
W20260923 20:09:27.733678 7055 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:33936: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41461/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260923 20:09:27.733733 7081 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 20:09:27.712605 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.190:42807 (local address 127.0.0.1:33936)
0923 20:09:27.713227 (+ 622us) negotiation.cc:107] Waiting for socket to connect
0923 20:09:27.713244 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0923 20:09:27.713361 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:09:27.713859 (+ 498us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:09:27.713862 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:09:27.713872 (+ 10us) client_negotiation.cc:189] Negotiated authn=JWT
0923 20:09:27.714195 (+ 323us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:09:27.714204 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:09:27.715642 (+ 1438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:09:27.715662 (+ 20us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:09:27.716734 (+ 1072us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:09:27.716744 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:09:27.717755 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:09:27.717760 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:09:27.717946 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:09:27.717985 (+ 39us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0923 20:09:27.733116 (+ 15131us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41461/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0923 20:09:27.733360 (+ 244us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.1.170.190:42807: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41461/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":463,"thread_start_us":346,"threads_started":1}
I20260923 20:09:27.734948 1706 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskqUuku6/build/asan/bin/kudu pid 6986
I20260923 20:09:27.736639 7047 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:09:27.832294 1706 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqUuku6/build/asan/bin/kudu with pid 6986
2026-09-23T20:09:27Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (760 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260923 20:09:27.887554 1706 mini_oidc.cc:152] Starting JWKS server
I20260923 20:09:27.889436 1706 webserver.cc:460] Webserver started at https://127.0.0.1:38677/ using document root <none> and password file <none>
I20260923 20:09:27.889686 1706 mini_oidc.cc:189] Starting OIDC Discovery server
I20260923 20:09:27.890341 1706 webserver.cc:460] Webserver started at http://127.0.0.1:45133/ using document root <none> and password file <none>
2026-09-23T20:09:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T20:09:27Z Disabled control of system clock
I20260923 20:09:27.918115 1706 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqUuku6/build/asan/bin/kudu
/tmp/dist-test-taskqUuku6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790194121577027-1706-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790194121577027-1706-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790194121577027-1706-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790194121577027-1706-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.170.190:36255
--webserver_interface=127.1.170.190
--webserver_port=0
--builtin_ntp_servers=127.1.170.148:38639
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.170.190:36255
--enable_jwt_token_auth=true
--jwks_url=https://localhost:38677/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260923 20:09:28.232553 7089 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:09:28.232872 7089 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:09:28.233044 7089 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:09:28.240491 7089 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 20:09:28.240600 7089 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:09:28.240687 7089 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:09:28.240798 7089 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 20:09:28.240878 7089 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260923 20:09:28.240978 7089 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260923 20:09:28.241055 7089 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:38677/jwks.json
I20260923 20:09:28.250336 7089 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.170.148:38639
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790194121577027-1706-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790194121577027-1706-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.170.190:36255
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.170.190:36255
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:38677/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790194121577027-1706-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.170.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790194121577027-1706-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:09:28.251654 7089 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:09:28.253522 7089 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:09:28.265461 7097 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:09:28.265471 7095 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:09:28.265537 7094 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:09:28.271677 7089 server_base.cc:1034] running on GCE node
I20260923 20:09:28.272529 7089 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:09:28.274081 7089 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:09:28.275323 7089 hybrid_clock.cc:648] HybridClock initialized: now 1790194168275293 us; error 48 us; skew 500 ppm
I20260923 20:09:28.275713 7089 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260923 20:09:28.278321 7089 webserver.cc:460] Webserver started at http://127.1.170.190:40285/ using document root <none> and password file <none>
I20260923 20:09:28.279073 7089 fs_manager.cc:359] Metadata directory not provided
I20260923 20:09:28.279219 7089 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:09:28.279551 7089 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:09:28.282487 7089 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790194121577027-1706-0/minicluster-data/master-0/data/instance:
uuid: "be3d8e30bbcf45288f140a05a1eff329"
format_stamp: "Formatted at 2026-09-23 20:09:28 on dist-test-slave-th4g"
server_key: "a5ada168db849adae65e190ef07ff8a1"
server_key_iv: "69962da9ad7f123d1b29677e9c7f8029"
server_key_version: "encryptionkey@0"
I20260923 20:09:28.283291 7089 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790194121577027-1706-0/minicluster-data/master-0/wal/instance:
uuid: "be3d8e30bbcf45288f140a05a1eff329"
format_stamp: "Formatted at 2026-09-23 20:09:28 on dist-test-slave-th4g"
server_key: "a5ada168db849adae65e190ef07ff8a1"
server_key_iv: "69962da9ad7f123d1b29677e9c7f8029"
server_key_version: "encryptionkey@0"
I20260923 20:09:28.288388 7089 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260923 20:09:28.293031 7103 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:09:28.294421 7089 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260923 20:09:28.294603 7089 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790194121577027-1706-0/minicluster-data/master-0/data,/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790194121577027-1706-0/minicluster-data/master-0/wal
uuid: "be3d8e30bbcf45288f140a05a1eff329"
format_stamp: "Formatted at 2026-09-23 20:09:28 on dist-test-slave-th4g"
server_key: "a5ada168db849adae65e190ef07ff8a1"
server_key_iv: "69962da9ad7f123d1b29677e9c7f8029"
server_key_version: "encryptionkey@0"
I20260923 20:09:28.294821 7089 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790194121577027-1706-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790194121577027-1706-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790194121577027-1706-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:09:28.306321 7089 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:09:28.307056 7089 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:09:28.307363 7089 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:09:28.325786 7089 rpc_server.cc:307] RPC server started. Bound to: 127.1.170.190:36255
I20260923 20:09:28.325806 7154 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.170.190:36255 every 8 connection(s)
I20260923 20:09:28.327239 7089 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790194121577027-1706-0/minicluster-data/master-0/data/info.pb
I20260923 20:09:28.331076 7155 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:09:28.337123 1706 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqUuku6/build/asan/bin/kudu as pid 7089
I20260923 20:09:28.337368 1706 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790194121577027-1706-0/minicluster-data/master-0/wal/instance
I20260923 20:09:28.338035 1706 external_mini_cluster.cc:1428] Setting key 8f878b42f1aeb0f0cc743324da55d28b
I20260923 20:09:28.342006 7155 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P be3d8e30bbcf45288f140a05a1eff329: Bootstrap starting.
I20260923 20:09:28.345257 7155 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P be3d8e30bbcf45288f140a05a1eff329: Neither blocks nor log segments found. Creating new log.
I20260923 20:09:28.346550 7155 log.cc:824] T 00000000000000000000000000000000 P be3d8e30bbcf45288f140a05a1eff329: Log is configured to *not* fsync() on all Append() calls
I20260923 20:09:28.349504 7155 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P be3d8e30bbcf45288f140a05a1eff329: No bootstrap required, opened a new log
I20260923 20:09:28.351959 7158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:09:28.340053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46902 (local address 127.1.170.190:36255)
0923 20:09:28.341066 (+ 1013us) server_negotiation.cc:206] Beginning negotiation
0923 20:09:28.341077 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0923 20:09:28.341117 (+ 40us) server_negotiation.cc:407] Connection header received
0923 20:09:28.342891 (+ 1774us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:09:28.342912 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:09:28.343351 (+ 439us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:09:28.343668 (+ 317us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:09:28.344240 (+ 572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:09:28.345545 (+ 1305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:09:28.347473 (+ 1928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:09:28.348162 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:09:28.348359 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:09:28.349518 (+ 1159us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:09:28.349527 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:09:28.349542 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 20:09:28.350225 (+ 683us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:09:28.350305 (+ 80us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 20:09:28.350329 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:09:28.350458 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0923 20:09:28.350689 (+ 231us) server_negotiation.cc:299] Negotiation successful
0923 20:09:28.350996 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":588,"thread_start_us":376,"threads_started":1}
I20260923 20:09:28.356110 1706 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 20:09:28.358310 7155 raft_consensus.cc:348] T 00000000000000000000000000000000 P be3d8e30bbcf45288f140a05a1eff329 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be3d8e30bbcf45288f140a05a1eff329" member_type: VOTER last_known_addr { host: "127.1.170.190" port: 36255 } }
I20260923 20:09:28.358757 7155 raft_consensus.cc:374] T 00000000000000000000000000000000 P be3d8e30bbcf45288f140a05a1eff329 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:09:28.358858 7155 raft_consensus.cc:729] T 00000000000000000000000000000000 P be3d8e30bbcf45288f140a05a1eff329 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: be3d8e30bbcf45288f140a05a1eff329, State: Initialized, Role: FOLLOWER
I20260923 20:09:28.359514 7155 consensus_queue.cc:260] T 00000000000000000000000000000000 P be3d8e30bbcf45288f140a05a1eff329 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be3d8e30bbcf45288f140a05a1eff329" member_type: VOTER last_known_addr { host: "127.1.170.190" port: 36255 } }
I20260923 20:09:28.359761 7155 raft_consensus.cc:388] T 00000000000000000000000000000000 P be3d8e30bbcf45288f140a05a1eff329 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 20:09:28.359861 7155 raft_consensus.cc:482] T 00000000000000000000000000000000 P be3d8e30bbcf45288f140a05a1eff329 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 20:09:28.359980 7155 raft_consensus.cc:3037] T 00000000000000000000000000000000 P be3d8e30bbcf45288f140a05a1eff329 [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:09:28.362110 7155 raft_consensus.cc:504] T 00000000000000000000000000000000 P be3d8e30bbcf45288f140a05a1eff329 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be3d8e30bbcf45288f140a05a1eff329" member_type: VOTER last_known_addr { host: "127.1.170.190" port: 36255 } }
I20260923 20:09:28.362546 7155 leader_election.cc:302] T 00000000000000000000000000000000 P be3d8e30bbcf45288f140a05a1eff329 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: be3d8e30bbcf45288f140a05a1eff329; no voters:
I20260923 20:09:28.363094 7155 leader_election.cc:288] T 00000000000000000000000000000000 P be3d8e30bbcf45288f140a05a1eff329 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 20:09:28.363189 7160 raft_consensus.cc:2781] T 00000000000000000000000000000000 P be3d8e30bbcf45288f140a05a1eff329 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 20:09:28.363965 7160 raft_consensus.cc:686] T 00000000000000000000000000000000 P be3d8e30bbcf45288f140a05a1eff329 [term 1 LEADER]: Becoming Leader. State: Replica: be3d8e30bbcf45288f140a05a1eff329, State: Running, Role: LEADER
I20260923 20:09:28.364578 7160 consensus_queue.cc:237] T 00000000000000000000000000000000 P be3d8e30bbcf45288f140a05a1eff329 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be3d8e30bbcf45288f140a05a1eff329" member_type: VOTER last_known_addr { host: "127.1.170.190" port: 36255 } }
I20260923 20:09:28.365271 7155 sys_catalog.cc:564] T 00000000000000000000000000000000 P be3d8e30bbcf45288f140a05a1eff329 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 20:09:28.367501 7162 sys_catalog.cc:455] T 00000000000000000000000000000000 P be3d8e30bbcf45288f140a05a1eff329 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "be3d8e30bbcf45288f140a05a1eff329" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be3d8e30bbcf45288f140a05a1eff329" member_type: VOTER last_known_addr { host: "127.1.170.190" port: 36255 } } }
I20260923 20:09:28.368168 7162 sys_catalog.cc:458] T 00000000000000000000000000000000 P be3d8e30bbcf45288f140a05a1eff329 [sys.catalog]: This master's current role is: LEADER
I20260923 20:09:28.369626 7161 sys_catalog.cc:455] T 00000000000000000000000000000000 P be3d8e30bbcf45288f140a05a1eff329 [sys.catalog]: SysCatalogTable state changed. Reason: New leader be3d8e30bbcf45288f140a05a1eff329. Latest consensus state: current_term: 1 leader_uuid: "be3d8e30bbcf45288f140a05a1eff329" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be3d8e30bbcf45288f140a05a1eff329" member_type: VOTER last_known_addr { host: "127.1.170.190" port: 36255 } } }
I20260923 20:09:28.370378 7161 sys_catalog.cc:458] T 00000000000000000000000000000000 P be3d8e30bbcf45288f140a05a1eff329 [sys.catalog]: This master's current role is: LEADER
I20260923 20:09:28.371383 7168 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 20:09:28.376205 7168 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 20:09:28.386791 7168 catalog_manager.cc:1347] Generated new cluster ID: bff5c59b4e044039b03f94f380d7166c
I20260923 20:09:28.386929 7168 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 20:09:28.404063 7168 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 20:09:28.405422 7168 catalog_manager.cc:1504] Loading token signing keys...
I20260923 20:09:28.414258 7168 catalog_manager.cc:5868] T 00000000000000000000000000000000 P be3d8e30bbcf45288f140a05a1eff329: Generated new TSK 0
I20260923 20:09:28.415364 7168 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 20:09:28.443532 7158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:09:28.435086 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46918 (local address 127.1.170.190:36255)
0923 20:09:28.435289 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0923 20:09:28.435296 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:09:28.435774 (+ 478us) server_negotiation.cc:407] Connection header received
0923 20:09:28.435991 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:09:28.435997 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:09:28.436102 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:09:28.436281 (+ 179us) server_negotiation.cc:226] Negotiated authn=JWT
0923 20:09:28.436685 (+ 404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:09:28.437809 (+ 1124us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:09:28.439205 (+ 1396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:09:28.439924 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:09:28.440055 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:09:28.440585 (+ 530us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0923 20:09:28.442873 (+ 2288us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38677/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0923 20:09:28.443200 (+ 327us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:46918: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38677/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":73}
W20260923 20:09:28.443666 7158 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:46918: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38677/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260923 20:09:28.445286 7184 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 20:09:28.434997 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.190:36255 (local address 127.0.0.1:46918)
0923 20:09:28.435674 (+ 677us) negotiation.cc:107] Waiting for socket to connect
0923 20:09:28.435691 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0923 20:09:28.435809 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:09:28.436289 (+ 480us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:09:28.436293 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:09:28.436302 (+ 9us) client_negotiation.cc:189] Negotiated authn=JWT
0923 20:09:28.436548 (+ 246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:09:28.436555 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:09:28.437989 (+ 1434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:09:28.437994 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:09:28.439061 (+ 1067us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:09:28.439071 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:09:28.440105 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:09:28.440110 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:09:28.440295 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:09:28.440330 (+ 35us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0923 20:09:28.443078 (+ 2748us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38677/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0923 20:09:28.444877 (+ 1799us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.1.170.190:36255: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38677/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":540,"mutex_wait_us":1507,"thread_start_us":389,"threads_started":1}
I20260923 20:09:28.446609 1706 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskqUuku6/build/asan/bin/kudu pid 7089
I20260923 20:09:28.447778 7150 generic_service.cc:196] Checking for leaks (request via RPC)
E20260923 20:09:28.458621 7185 webserver.cc:533] Webserver: sslize failed:
E20260923 20:09:28.458778 7185 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260923 20:09:28.541224 1706 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqUuku6/build/asan/bin/kudu with pid 7089
2026-09-23T20:09:28Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (709 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-09-23T20:09:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T20:09:28Z Disabled control of system clock
I20260923 20:09:28.627066 1706 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqUuku6/build/asan/bin/kudu
/tmp/dist-test-taskqUuku6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790194121577027-1706-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790194121577027-1706-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790194121577027-1706-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790194121577027-1706-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.170.190:36173
--webserver_interface=127.1.170.190
--webserver_port=0
--builtin_ntp_servers=127.1.170.148:40003
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.170.190:36173
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790194121577027-1706-0/testchainca.cert with env {}
W20260923 20:09:28.962101 7190 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:09:28.962443 7190 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:09:28.962533 7190 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:09:28.971084 7190 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 20:09:28.971217 7190 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:09:28.971256 7190 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:09:28.971310 7190 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 20:09:28.971367 7190 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260923 20:09:28.971431 7190 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260923 20:09:28.971470 7190 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260923 20:09:28.981834 7190 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.170.148:40003
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790194121577027-1706-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790194121577027-1706-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.170.190:36173
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.170.190:36173
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790194121577027-1706-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.170.190
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790194121577027-1706-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790194121577027-1706-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:09:28.983212 7190 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:09:28.984508 7190 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:09:28.999094 7195 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:09:28.999181 7198 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:09:28.999413 7196 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:09:29.001055 7190 server_base.cc:1034] running on GCE node
I20260923 20:09:29.002142 7190 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:09:29.003830 7190 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:09:29.005205 7190 hybrid_clock.cc:648] HybridClock initialized: now 1790194169005119 us; error 51 us; skew 500 ppm
I20260923 20:09:29.005647 7190 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260923 20:09:29.008406 7190 webserver.cc:460] Webserver started at http://127.1.170.190:33441/ using document root <none> and password file <none>
I20260923 20:09:29.009167 7190 fs_manager.cc:359] Metadata directory not provided
I20260923 20:09:29.009310 7190 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:09:29.009647 7190 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:09:29.012737 7190 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790194121577027-1706-0/minicluster-data/master-0/data/instance:
uuid: "46a70a6550b94cf79e2076f5c1feb026"
format_stamp: "Formatted at 2026-09-23 20:09:29 on dist-test-slave-th4g"
server_key: "2a61345fe90325f44cbf8fb41a60dd45"
server_key_iv: "b55015a304eb9a87f90d03726a26b294"
server_key_version: "encryptionkey@0"
I20260923 20:09:29.013628 7190 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790194121577027-1706-0/minicluster-data/master-0/wal/instance:
uuid: "46a70a6550b94cf79e2076f5c1feb026"
format_stamp: "Formatted at 2026-09-23 20:09:29 on dist-test-slave-th4g"
server_key: "2a61345fe90325f44cbf8fb41a60dd45"
server_key_iv: "b55015a304eb9a87f90d03726a26b294"
server_key_version: "encryptionkey@0"
I20260923 20:09:29.019176 7190 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260923 20:09:29.022967 7204 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:09:29.024448 7190 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260923 20:09:29.024663 7190 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790194121577027-1706-0/minicluster-data/master-0/data,/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790194121577027-1706-0/minicluster-data/master-0/wal
uuid: "46a70a6550b94cf79e2076f5c1feb026"
format_stamp: "Formatted at 2026-09-23 20:09:29 on dist-test-slave-th4g"
server_key: "2a61345fe90325f44cbf8fb41a60dd45"
server_key_iv: "b55015a304eb9a87f90d03726a26b294"
server_key_version: "encryptionkey@0"
I20260923 20:09:29.024896 7190 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790194121577027-1706-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790194121577027-1706-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790194121577027-1706-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:09:29.043288 7190 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:09:29.044157 7190 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:09:29.044459 7190 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:09:29.063868 7190 rpc_server.cc:307] RPC server started. Bound to: 127.1.170.190:36173
I20260923 20:09:29.063889 7255 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.170.190:36173 every 8 connection(s)
I20260923 20:09:29.065274 7190 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790194121577027-1706-0/minicluster-data/master-0/data/info.pb
I20260923 20:09:29.069308 1706 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqUuku6/build/asan/bin/kudu as pid 7190
I20260923 20:09:29.069531 1706 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790194121577027-1706-0/minicluster-data/master-0/wal/instance
I20260923 20:09:29.069531 7256 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:09:29.070343 1706 external_mini_cluster.cc:1428] Setting key 004b1e75c3290fde6695a59e304af76f
I20260923 20:09:29.086295 7256 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 46a70a6550b94cf79e2076f5c1feb026: Bootstrap starting.
I20260923 20:09:29.089108 7259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:09:29.072633 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33982 (local address 127.1.170.190:36173)
0923 20:09:29.074178 (+ 1545us) server_negotiation.cc:206] Beginning negotiation
0923 20:09:29.074205 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0923 20:09:29.074517 (+ 312us) server_negotiation.cc:407] Connection header received
0923 20:09:29.077355 (+ 2838us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:09:29.077396 (+ 41us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:09:29.078406 (+ 1010us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:09:29.079122 (+ 716us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:09:29.079741 (+ 619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:09:29.081622 (+ 1881us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:09:29.083467 (+ 1845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:09:29.084591 (+ 1124us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:09:29.084793 (+ 202us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:09:29.086110 (+ 1317us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:09:29.086121 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:09:29.086142 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 20:09:29.086894 (+ 752us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:09:29.087006 (+ 112us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 20:09:29.087037 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:09:29.087205 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0923 20:09:29.087409 (+ 204us) server_negotiation.cc:299] Negotiation successful
0923 20:09:29.087762 (+ 353us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":755,"thread_start_us":462,"threads_started":1}
I20260923 20:09:29.090361 7256 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 46a70a6550b94cf79e2076f5c1feb026: Neither blocks nor log segments found. Creating new log.
I20260923 20:09:29.092181 7256 log.cc:824] T 00000000000000000000000000000000 P 46a70a6550b94cf79e2076f5c1feb026: Log is configured to *not* fsync() on all Append() calls
I20260923 20:09:29.094625 1706 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 20:09:29.096783 7256 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 46a70a6550b94cf79e2076f5c1feb026: No bootstrap required, opened a new log
I20260923 20:09:29.105273 7256 raft_consensus.cc:348] T 00000000000000000000000000000000 P 46a70a6550b94cf79e2076f5c1feb026 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46a70a6550b94cf79e2076f5c1feb026" member_type: VOTER last_known_addr { host: "127.1.170.190" port: 36173 } }
I20260923 20:09:29.106037 7256 raft_consensus.cc:374] T 00000000000000000000000000000000 P 46a70a6550b94cf79e2076f5c1feb026 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:09:29.106135 7256 raft_consensus.cc:729] T 00000000000000000000000000000000 P 46a70a6550b94cf79e2076f5c1feb026 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46a70a6550b94cf79e2076f5c1feb026, State: Initialized, Role: FOLLOWER
I20260923 20:09:29.106731 7256 consensus_queue.cc:260] T 00000000000000000000000000000000 P 46a70a6550b94cf79e2076f5c1feb026 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46a70a6550b94cf79e2076f5c1feb026" member_type: VOTER last_known_addr { host: "127.1.170.190" port: 36173 } }
I20260923 20:09:29.106992 7256 raft_consensus.cc:388] T 00000000000000000000000000000000 P 46a70a6550b94cf79e2076f5c1feb026 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 20:09:29.107141 7256 raft_consensus.cc:482] T 00000000000000000000000000000000 P 46a70a6550b94cf79e2076f5c1feb026 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 20:09:29.107401 7256 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 46a70a6550b94cf79e2076f5c1feb026 [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:09:29.109597 7256 raft_consensus.cc:504] T 00000000000000000000000000000000 P 46a70a6550b94cf79e2076f5c1feb026 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46a70a6550b94cf79e2076f5c1feb026" member_type: VOTER last_known_addr { host: "127.1.170.190" port: 36173 } }
I20260923 20:09:29.110219 7256 leader_election.cc:302] T 00000000000000000000000000000000 P 46a70a6550b94cf79e2076f5c1feb026 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 46a70a6550b94cf79e2076f5c1feb026; no voters:
I20260923 20:09:29.110899 7256 leader_election.cc:288] T 00000000000000000000000000000000 P 46a70a6550b94cf79e2076f5c1feb026 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 20:09:29.111084 7261 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 46a70a6550b94cf79e2076f5c1feb026 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 20:09:29.112481 7261 raft_consensus.cc:686] T 00000000000000000000000000000000 P 46a70a6550b94cf79e2076f5c1feb026 [term 1 LEADER]: Becoming Leader. State: Replica: 46a70a6550b94cf79e2076f5c1feb026, State: Running, Role: LEADER
I20260923 20:09:29.113001 7261 consensus_queue.cc:237] T 00000000000000000000000000000000 P 46a70a6550b94cf79e2076f5c1feb026 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46a70a6550b94cf79e2076f5c1feb026" member_type: VOTER last_known_addr { host: "127.1.170.190" port: 36173 } }
I20260923 20:09:29.113221 7256 sys_catalog.cc:564] T 00000000000000000000000000000000 P 46a70a6550b94cf79e2076f5c1feb026 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 20:09:29.116074 7263 sys_catalog.cc:455] T 00000000000000000000000000000000 P 46a70a6550b94cf79e2076f5c1feb026 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 46a70a6550b94cf79e2076f5c1feb026. Latest consensus state: current_term: 1 leader_uuid: "46a70a6550b94cf79e2076f5c1feb026" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46a70a6550b94cf79e2076f5c1feb026" member_type: VOTER last_known_addr { host: "127.1.170.190" port: 36173 } } }
I20260923 20:09:29.116199 7262 sys_catalog.cc:455] T 00000000000000000000000000000000 P 46a70a6550b94cf79e2076f5c1feb026 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "46a70a6550b94cf79e2076f5c1feb026" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46a70a6550b94cf79e2076f5c1feb026" member_type: VOTER last_known_addr { host: "127.1.170.190" port: 36173 } } }
I20260923 20:09:29.116549 7263 sys_catalog.cc:458] T 00000000000000000000000000000000 P 46a70a6550b94cf79e2076f5c1feb026 [sys.catalog]: This master's current role is: LEADER
I20260923 20:09:29.116624 7262 sys_catalog.cc:458] T 00000000000000000000000000000000 P 46a70a6550b94cf79e2076f5c1feb026 [sys.catalog]: This master's current role is: LEADER
I20260923 20:09:29.117612 7268 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 20:09:29.126037 7268 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 20:09:29.138711 7268 catalog_manager.cc:1347] Generated new cluster ID: 1afe8d121994448ab1602e020b8c61ae
I20260923 20:09:29.138851 7268 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 20:09:29.153254 7268 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 20:09:29.155159 7268 catalog_manager.cc:1504] Loading token signing keys...
I20260923 20:09:29.166975 7268 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 46a70a6550b94cf79e2076f5c1feb026: Generated new TSK 0
I20260923 20:09:29.168217 7268 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 20:09:29.271697 7259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:09:29.189590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33988 (local address 127.1.170.190:36173)
0923 20:09:29.189772 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0923 20:09:29.189777 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 20:09:29.190358 (+ 581us) server_negotiation.cc:407] Connection header received
0923 20:09:29.190581 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:09:29.190585 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:09:29.190662 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:09:29.190838 (+ 176us) server_negotiation.cc:226] Negotiated authn=JWT
0923 20:09:29.191397 (+ 559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:09:29.192491 (+ 1094us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:09:29.194692 (+ 2201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:09:29.195195 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:09:29.195317 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:09:29.195934 (+ 617us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0923 20:09:29.270953 (+ 75019us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0923 20:09:29.271385 (+ 432us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:33988: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":61}
W20260923 20:09:29.271842 7259 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:33988: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260923 20:09:29.271970 7285 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 20:09:29.189336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.190:36173 (local address 127.0.0.1:33988)
0923 20:09:29.190215 (+ 879us) negotiation.cc:107] Waiting for socket to connect
0923 20:09:29.190237 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0923 20:09:29.190412 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:09:29.190853 (+ 441us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:09:29.190859 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:09:29.190872 (+ 13us) client_negotiation.cc:189] Negotiated authn=JWT
0923 20:09:29.191211 (+ 339us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:09:29.191223 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:09:29.192662 (+ 1439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:09:29.192668 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:09:29.194445 (+ 1777us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:09:29.194462 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:09:29.195391 (+ 929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:09:29.195398 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:09:29.195590 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:09:29.195648 (+ 58us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0923 20:09:29.271328 (+ 75680us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0923 20:09:29.271569 (+ 241us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.1.170.190:36173: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":606,"thread_start_us":427,"threads_started":1}
I20260923 20:09:29.273113 1706 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskqUuku6/build/asan/bin/kudu pid 7190
I20260923 20:09:29.274591 7251 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:09:29.378607 1706 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqUuku6/build/asan/bin/kudu with pid 7190
2026-09-23T20:09:29Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (816 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (440 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (1517 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790194121577027-1706-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 20:09:31 dist-test-slave-th4g krb5kdc[7301](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 20:09:31 dist-test-slave-th4g krb5kdc[7301](info): set up 2 sockets
Sep 23 20:09:31 dist-test-slave-th4g krb5kdc[7301](info): commencing operation
krb5kdc: starting...
W20260923 20:09:32.434705 1706 mini_kdc.cc:121] Time spent starting KDC: real 1.062s user 0.003s sys 0.015s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790194121577027-1706-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790194121577027-1706-0/krb5kdc/test-user.keytab.
Sep 23 20:09:32 dist-test-slave-th4g krb5kdc[7301](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194172, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T20:09:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T20:09:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.170.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.170.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.170.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790194121577027-1706-0/krb5kdc/kudu_127.1.170.190.keytab.
Entry for principal kudu/127.1.170.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790194121577027-1706-0/krb5kdc/kudu_127.1.170.190.keytab.
I20260923 20:09:32.645784 1706 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqUuku6/build/asan/bin/kudu
/tmp/dist-test-taskqUuku6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790194121577027-1706-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790194121577027-1706-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790194121577027-1706-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790194121577027-1706-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.170.190:43197
--webserver_interface=127.1.170.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790194121577027-1706-0/krb5kdc/kudu_127.1.170.190.keytab
--principal=kudu/127.1.170.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.170.148:43289
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.170.190:43197
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790194121577027-1706-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790194121577027-1706-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790194121577027-1706-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:09:32.971781 7314 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:09:32.972209 7314 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:09:32.972383 7314 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:09:32.979815 7314 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 20:09:32.979996 7314 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:09:32.980110 7314 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:09:32.980232 7314 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 20:09:32.980317 7314 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 20:09:32.990356 7314 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.170.148:43289
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790194121577027-1706-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790194121577027-1706-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.170.190:43197
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790194121577027-1706-0/krb5kdc/kudu_127.1.170.190.keytab
--principal=kudu/127.1.170.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.170.190:43197
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790194121577027-1706-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.170.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790194121577027-1706-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:09:32.991786 7314 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:09:32.993129 7314 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:09:33.005473 7322 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:09:33.005513 7320 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:09:33.005434 7319 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:09:33.006397 7314 server_base.cc:1034] running on GCE node
I20260923 20:09:33.007248 7314 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:09:33.008764 7314 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:09:33.010025 7314 hybrid_clock.cc:648] HybridClock initialized: now 1790194173009973 us; error 44 us; skew 500 ppm
I20260923 20:09:33.010489 7314 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:09:33 dist-test-slave-th4g krb5kdc[7301](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194173, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.170.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:09:33.014209 7314 init.cc:377] Logged in from keytab as kudu/127.1.170.190@KRBTEST.COM (short username kudu)
I20260923 20:09:33.015901 7314 webserver.cc:460] Webserver started at http://127.1.170.190:43257/ using document root <none> and password file <none>
I20260923 20:09:33.016638 7314 fs_manager.cc:359] Metadata directory not provided
I20260923 20:09:33.016793 7314 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:09:33.017127 7314 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:09:33.020313 7314 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790194121577027-1706-0/minicluster-data/master-0/data/instance:
uuid: "2c3952c9affd4641b0af4ca3ab3f8d82"
format_stamp: "Formatted at 2026-09-23 20:09:33 on dist-test-slave-th4g"
server_key: "36efc15824d9771197a3c7d8c044ee32"
server_key_iv: "c885b75762395d9395d4721ee0330508"
server_key_version: "encryptionkey@0"
I20260923 20:09:33.021166 7314 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790194121577027-1706-0/minicluster-data/master-0/wal/instance:
uuid: "2c3952c9affd4641b0af4ca3ab3f8d82"
format_stamp: "Formatted at 2026-09-23 20:09:33 on dist-test-slave-th4g"
server_key: "36efc15824d9771197a3c7d8c044ee32"
server_key_iv: "c885b75762395d9395d4721ee0330508"
server_key_version: "encryptionkey@0"
I20260923 20:09:33.026818 7314 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260923 20:09:33.030507 7329 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:09:33.031999 7314 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260923 20:09:33.032225 7314 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790194121577027-1706-0/minicluster-data/master-0/data,/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790194121577027-1706-0/minicluster-data/master-0/wal
uuid: "2c3952c9affd4641b0af4ca3ab3f8d82"
format_stamp: "Formatted at 2026-09-23 20:09:33 on dist-test-slave-th4g"
server_key: "36efc15824d9771197a3c7d8c044ee32"
server_key_iv: "c885b75762395d9395d4721ee0330508"
server_key_version: "encryptionkey@0"
I20260923 20:09:33.032449 7314 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790194121577027-1706-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790194121577027-1706-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790194121577027-1706-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:09:33.051573 7314 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:09:33.056783 7314 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:09:33.057094 7314 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:09:33.079676 7314 rpc_server.cc:307] RPC server started. Bound to: 127.1.170.190:43197
I20260923 20:09:33.079699 7380 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.170.190:43197 every 8 connection(s)
I20260923 20:09:33.081061 7314 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790194121577027-1706-0/minicluster-data/master-0/data/info.pb
I20260923 20:09:33.084810 7381 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:09:33.087039 1706 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqUuku6/build/asan/bin/kudu as pid 7314
I20260923 20:09:33.087376 1706 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790194121577027-1706-0/minicluster-data/master-0/wal/instance
I20260923 20:09:33.088267 1706 external_mini_cluster.cc:1428] Setting key 1cc5eb720ef35d3bbd89edf2ea6ec418
I20260923 20:09:33.096655 7381 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2c3952c9affd4641b0af4ca3ab3f8d82: Bootstrap starting.
I20260923 20:09:33.099951 7381 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2c3952c9affd4641b0af4ca3ab3f8d82: Neither blocks nor log segments found. Creating new log.
I20260923 20:09:33.101022 7381 log.cc:824] T 00000000000000000000000000000000 P 2c3952c9affd4641b0af4ca3ab3f8d82: Log is configured to *not* fsync() on all Append() calls
Sep 23 20:09:33 dist-test-slave-th4g krb5kdc[7301](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194172, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.170.190@KRBTEST.COM
I20260923 20:09:33.104147 7381 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2c3952c9affd4641b0af4ca3ab3f8d82: No bootstrap required, opened a new log
I20260923 20:09:33.111865 7384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:09:33.090601 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46940 (local address 127.1.170.190:43197)
0923 20:09:33.091437 (+ 836us) server_negotiation.cc:206] Beginning negotiation
0923 20:09:33.091451 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0923 20:09:33.091493 (+ 42us) server_negotiation.cc:407] Connection header received
0923 20:09:33.093242 (+ 1749us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:09:33.093267 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:09:33.093706 (+ 439us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:09:33.094208 (+ 502us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:09:33.095612 (+ 1404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:09:33.096938 (+ 1326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:09:33.098267 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:09:33.098959 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:09:33.099165 (+ 206us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:09:33.103853 (+ 4688us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:09:33.103861 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:09:33.103876 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:09:33.103916 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:09:33.107460 (+ 3544us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:09:33.108272 (+ 812us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:09:33.108283 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:09:33.108290 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:09:33.108470 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:09:33.108881 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:09:33.108885 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:09:33.108887 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:09:33.109483 (+ 596us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:09:33.109815 (+ 332us) server_negotiation.cc:1035] Waiting for connection context
0923 20:09:33.110324 (+ 509us) server_negotiation.cc:299] Negotiation successful
0923 20:09:33.110767 (+ 443us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":607,"thread_start_us":376,"threads_started":1}
I20260923 20:09:33.112351 7381 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2c3952c9affd4641b0af4ca3ab3f8d82 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c3952c9affd4641b0af4ca3ab3f8d82" member_type: VOTER last_known_addr { host: "127.1.170.190" port: 43197 } }
I20260923 20:09:33.112731 7381 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2c3952c9affd4641b0af4ca3ab3f8d82 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:09:33.112861 7381 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2c3952c9affd4641b0af4ca3ab3f8d82 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2c3952c9affd4641b0af4ca3ab3f8d82, State: Initialized, Role: FOLLOWER
I20260923 20:09:33.113456 7381 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2c3952c9affd4641b0af4ca3ab3f8d82 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c3952c9affd4641b0af4ca3ab3f8d82" member_type: VOTER last_known_addr { host: "127.1.170.190" port: 43197 } }
I20260923 20:09:33.113715 7381 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2c3952c9affd4641b0af4ca3ab3f8d82 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 20:09:33.113869 7381 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2c3952c9affd4641b0af4ca3ab3f8d82 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 20:09:33.114123 7381 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2c3952c9affd4641b0af4ca3ab3f8d82 [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:09:33.116648 7381 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2c3952c9affd4641b0af4ca3ab3f8d82 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c3952c9affd4641b0af4ca3ab3f8d82" member_type: VOTER last_known_addr { host: "127.1.170.190" port: 43197 } }
I20260923 20:09:33.117228 7381 leader_election.cc:302] T 00000000000000000000000000000000 P 2c3952c9affd4641b0af4ca3ab3f8d82 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2c3952c9affd4641b0af4ca3ab3f8d82; no voters:
I20260923 20:09:33.117856 7381 leader_election.cc:288] T 00000000000000000000000000000000 P 2c3952c9affd4641b0af4ca3ab3f8d82 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 20:09:33.118239 7386 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2c3952c9affd4641b0af4ca3ab3f8d82 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 20:09:33.118901 7386 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2c3952c9affd4641b0af4ca3ab3f8d82 [term 1 LEADER]: Becoming Leader. State: Replica: 2c3952c9affd4641b0af4ca3ab3f8d82, State: Running, Role: LEADER
I20260923 20:09:33.119428 7386 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2c3952c9affd4641b0af4ca3ab3f8d82 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c3952c9affd4641b0af4ca3ab3f8d82" member_type: VOTER last_known_addr { host: "127.1.170.190" port: 43197 } }
I20260923 20:09:33.119992 7381 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2c3952c9affd4641b0af4ca3ab3f8d82 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 20:09:33.122223 7388 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2c3952c9affd4641b0af4ca3ab3f8d82 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2c3952c9affd4641b0af4ca3ab3f8d82. Latest consensus state: current_term: 1 leader_uuid: "2c3952c9affd4641b0af4ca3ab3f8d82" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c3952c9affd4641b0af4ca3ab3f8d82" member_type: VOTER last_known_addr { host: "127.1.170.190" port: 43197 } } }
I20260923 20:09:33.122676 7388 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2c3952c9affd4641b0af4ca3ab3f8d82 [sys.catalog]: This master's current role is: LEADER
I20260923 20:09:33.122169 7387 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2c3952c9affd4641b0af4ca3ab3f8d82 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2c3952c9affd4641b0af4ca3ab3f8d82" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c3952c9affd4641b0af4ca3ab3f8d82" member_type: VOTER last_known_addr { host: "127.1.170.190" port: 43197 } } }
I20260923 20:09:33.123121 7387 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2c3952c9affd4641b0af4ca3ab3f8d82 [sys.catalog]: This master's current role is: LEADER
I20260923 20:09:33.123512 7394 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 20:09:33.130226 7394 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 20:09:33.153867 7394 catalog_manager.cc:1347] Generated new cluster ID: 30d86df6198143d89429da14c95bbd85
I20260923 20:09:33.154023 7394 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 20:09:33.170307 7394 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 20:09:33.172222 7394 catalog_manager.cc:1504] Loading token signing keys...
I20260923 20:09:33.181519 7394 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2c3952c9affd4641b0af4ca3ab3f8d82: Generated new TSK 0
I20260923 20:09:33.182578 7394 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.170.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.170.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.170.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790194121577027-1706-0/krb5kdc/kudu_127.1.170.129.keytab.
Entry for principal kudu/127.1.170.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790194121577027-1706-0/krb5kdc/kudu_127.1.170.129.keytab.
I20260923 20:09:33.254424 1706 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqUuku6/build/asan/bin/kudu
/tmp/dist-test-taskqUuku6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790194121577027-1706-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790194121577027-1706-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790194121577027-1706-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790194121577027-1706-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.170.129:0
--local_ip_for_outbound_sockets=127.1.170.129
--webserver_interface=127.1.170.129
--webserver_port=0
--tserver_master_addrs=127.1.170.190:43197
--keytab_file=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790194121577027-1706-0/krb5kdc/kudu_127.1.170.129.keytab
--principal=kudu/127.1.170.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.170.148:43289
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790194121577027-1706-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790194121577027-1706-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790194121577027-1706-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:09:33.622133 7407 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:09:33.622438 7407 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:09:33.622532 7407 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:09:33.630280 7407 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:09:33.630400 7407 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:09:33.630514 7407 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.170.129
I20260923 20:09:33.640784 7407 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.170.148:43289
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790194121577027-1706-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790194121577027-1706-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790194121577027-1706-0/krb5kdc/kudu_127.1.170.129.keytab
--principal=kudu/127.1.170.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.170.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790194121577027-1706-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.170.129
--webserver_port=0
--tserver_master_addrs=127.1.170.190:43197
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.170.129
--log_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790194121577027-1706-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:09:33.642150 7407 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:09:33.643424 7407 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:09:33.657723 7412 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:09:33.657732 7413 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:09:33.657761 7415 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:09:33.659003 7407 server_base.cc:1034] running on GCE node
I20260923 20:09:33.659657 7407 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:09:33.660713 7407 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:09:33.661988 7407 hybrid_clock.cc:648] HybridClock initialized: now 1790194173661935 us; error 45 us; skew 500 ppm
I20260923 20:09:33.662312 7407 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:09:33 dist-test-slave-th4g krb5kdc[7301](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194173, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.170.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:09:33.665531 7407 init.cc:377] Logged in from keytab as kudu/127.1.170.129@KRBTEST.COM (short username kudu)
I20260923 20:09:33.667351 7407 webserver.cc:460] Webserver started at http://127.1.170.129:35247/ using document root <none> and password file <none>
I20260923 20:09:33.668015 7407 fs_manager.cc:359] Metadata directory not provided
I20260923 20:09:33.668118 7407 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:09:33.668356 7407 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:09:33.671568 7407 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790194121577027-1706-0/minicluster-data/ts-0/data/instance:
uuid: "a42ef9bcf13b46cab3c038f017df2d4d"
format_stamp: "Formatted at 2026-09-23 20:09:33 on dist-test-slave-th4g"
server_key: "f7228ec98716d96d57edf8f4e5dfff36"
server_key_iv: "57318738fd7fc9009bae1752da30c06d"
server_key_version: "encryptionkey@0"
I20260923 20:09:33.672353 7407 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790194121577027-1706-0/minicluster-data/ts-0/wal/instance:
uuid: "a42ef9bcf13b46cab3c038f017df2d4d"
format_stamp: "Formatted at 2026-09-23 20:09:33 on dist-test-slave-th4g"
server_key: "f7228ec98716d96d57edf8f4e5dfff36"
server_key_iv: "57318738fd7fc9009bae1752da30c06d"
server_key_version: "encryptionkey@0"
I20260923 20:09:33.677964 7407 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260923 20:09:33.681641 7422 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:09:33.683135 7407 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260923 20:09:33.683357 7407 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790194121577027-1706-0/minicluster-data/ts-0/data,/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790194121577027-1706-0/minicluster-data/ts-0/wal
uuid: "a42ef9bcf13b46cab3c038f017df2d4d"
format_stamp: "Formatted at 2026-09-23 20:09:33 on dist-test-slave-th4g"
server_key: "f7228ec98716d96d57edf8f4e5dfff36"
server_key_iv: "57318738fd7fc9009bae1752da30c06d"
server_key_version: "encryptionkey@0"
I20260923 20:09:33.683558 7407 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790194121577027-1706-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790194121577027-1706-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790194121577027-1706-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:09:33.699088 7407 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:09:33.703634 7407 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:09:33.703935 7407 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:09:33.704943 7407 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 20:09:33.706329 7407 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 20:09:33.706398 7407 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 20:09:33.706537 7407 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 20:09:33.706602 7407 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 20:09:33.741525 7407 rpc_server.cc:307] RPC server started. Bound to: 127.1.170.129:40137
I20260923 20:09:33.741569 7534 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.170.129:40137 every 8 connection(s)
I20260923 20:09:33.742846 7407 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790194121577027-1706-0/minicluster-data/ts-0/data/info.pb
I20260923 20:09:33.750881 1706 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqUuku6/build/asan/bin/kudu as pid 7407
I20260923 20:09:33.751097 1706 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790194121577027-1706-0/minicluster-data/ts-0/wal/instance
I20260923 20:09:33.751874 1706 external_mini_cluster.cc:1428] Setting key dd08a4e3ad3cf3477dc7d2decff5d51c
Sep 23 20:09:33 dist-test-slave-th4g krb5kdc[7301](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194173, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.170.129@KRBTEST.COM for kudu/127.1.170.190@KRBTEST.COM
I20260923 20:09:33.769924 7537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:09:33.745737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.170.129:49871 (local address 127.1.170.190:43197)
0923 20:09:33.746390 (+ 653us) server_negotiation.cc:206] Beginning negotiation
0923 20:09:33.746395 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 20:09:33.747749 (+ 1354us) server_negotiation.cc:407] Connection header received
0923 20:09:33.749757 (+ 2008us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:09:33.749762 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:09:33.749856 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:09:33.750027 (+ 171us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:09:33.751803 (+ 1776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:09:33.753119 (+ 1316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:09:33.755219 (+ 2100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:09:33.756261 (+ 1042us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:09:33.756437 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:09:33.762630 (+ 6193us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:09:33.762641 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:09:33.762649 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:09:33.762702 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:09:33.766301 (+ 3599us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:09:33.767161 (+ 860us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:09:33.767170 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:09:33.767176 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:09:33.767338 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:09:33.767807 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:09:33.767814 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:09:33.767820 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:09:33.768216 (+ 396us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:09:33.768365 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0923 20:09:33.769103 (+ 738us) server_negotiation.cc:299] Negotiation successful
0923 20:09:33.769366 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":535,"spinlock_wait_cycles":1792,"thread_start_us":479,"threads_started":1}
I20260923 20:09:33.770681 7538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:09:33.746453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.190:43197 (local address 127.1.170.129:49871)
0923 20:09:33.747547 (+ 1094us) negotiation.cc:107] Waiting for socket to connect
0923 20:09:33.747595 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0923 20:09:33.749437 (+ 1842us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:09:33.750300 (+ 863us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:09:33.750309 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:09:33.750890 (+ 581us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:09:33.751598 (+ 708us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:09:33.751617 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:09:33.753328 (+ 1711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:09:33.753333 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:09:33.754922 (+ 1589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:09:33.754942 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:09:33.756634 (+ 1692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:09:33.756645 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:09:33.757268 (+ 623us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:09:33.758450 (+ 1182us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:09:33.758518 (+ 68us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:09:33.762275 (+ 3757us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:09:33.766502 (+ 4227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:09:33.766511 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:09:33.766527 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:09:33.766994 (+ 467us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:09:33.767461 (+ 467us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:09:33.767465 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:09:33.767468 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:09:33.767676 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:09:33.768352 (+ 676us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:09:33.768358 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:09:33.768706 (+ 348us) client_negotiation.cc:769] Sending connection context
0923 20:09:33.768940 (+ 234us) client_negotiation.cc:240] Negotiation successful
0923 20:09:33.769398 (+ 458us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":810,"thread_start_us":561,"threads_started":1}
I20260923 20:09:33.772588 7535 heartbeater.cc:347] Connected to a master server at 127.1.170.190:43197
I20260923 20:09:33.772912 7535 heartbeater.cc:464] Registering TS with master...
I20260923 20:09:33.773658 7535 heartbeater.cc:511] Master 127.1.170.190:43197 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.1.170.130@KRBTEST.COM; defaulting to no policy
I20260923 20:09:33.775988 7346 ts_manager.cc:194] Registered new tserver with Master: a42ef9bcf13b46cab3c038f017df2d4d (127.1.170.129:40137)
I20260923 20:09:33.777431 7346 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.170.129@KRBTEST.COM'} at 127.1.170.129:49871
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.170.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.170.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790194121577027-1706-0/krb5kdc/kudu_127.1.170.130.keytab.
Entry for principal kudu/127.1.170.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790194121577027-1706-0/krb5kdc/kudu_127.1.170.130.keytab.
I20260923 20:09:33.805207 1706 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqUuku6/build/asan/bin/kudu
/tmp/dist-test-taskqUuku6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790194121577027-1706-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790194121577027-1706-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790194121577027-1706-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790194121577027-1706-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.170.130:0
--local_ip_for_outbound_sockets=127.1.170.130
--webserver_interface=127.1.170.130
--webserver_port=0
--tserver_master_addrs=127.1.170.190:43197
--keytab_file=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790194121577027-1706-0/krb5kdc/kudu_127.1.170.130.keytab
--principal=kudu/127.1.170.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.170.148:43289
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790194121577027-1706-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790194121577027-1706-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790194121577027-1706-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:09:34.156512 7541 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:09:34.156813 7541 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:09:34.156912 7541 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:09:34.166326 7541 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:09:34.166473 7541 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:09:34.166591 7541 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.170.130
I20260923 20:09:34.178591 7541 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.170.148:43289
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790194121577027-1706-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790194121577027-1706-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790194121577027-1706-0/krb5kdc/kudu_127.1.170.130.keytab
--principal=kudu/127.1.170.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.170.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790194121577027-1706-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.170.130
--webserver_port=0
--tserver_master_addrs=127.1.170.190:43197
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.170.130
--log_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790194121577027-1706-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:09:34.179916 7541 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:09:34.181181 7541 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:09:34.195406 7547 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:09:34.195444 7549 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:09:34.195778 7546 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:09:34.196838 7541 server_base.cc:1034] running on GCE node
I20260923 20:09:34.197551 7541 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:09:34.198774 7541 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:09:34.200047 7541 hybrid_clock.cc:648] HybridClock initialized: now 1790194174199977 us; error 63 us; skew 500 ppm
I20260923 20:09:34.200423 7541 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:09:34 dist-test-slave-th4g krb5kdc[7301](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194174, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.170.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:09:34.204551 7541 init.cc:377] Logged in from keytab as kudu/127.1.170.130@KRBTEST.COM (short username kudu)
I20260923 20:09:34.206449 7541 webserver.cc:460] Webserver started at http://127.1.170.130:41379/ using document root <none> and password file <none>
I20260923 20:09:34.207190 7541 fs_manager.cc:359] Metadata directory not provided
I20260923 20:09:34.207352 7541 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:09:34.207671 7541 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:09:34.210778 7541 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790194121577027-1706-0/minicluster-data/ts-1/data/instance:
uuid: "b2f17c13664541b6abee0b077b094413"
format_stamp: "Formatted at 2026-09-23 20:09:34 on dist-test-slave-th4g"
server_key: "12f01e1f6d0d0154e92a79d1431b13c9"
server_key_iv: "76c27d6fa37b1ed0aa68d51a195bc32f"
server_key_version: "encryptionkey@0"
I20260923 20:09:34.211604 7541 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790194121577027-1706-0/minicluster-data/ts-1/wal/instance:
uuid: "b2f17c13664541b6abee0b077b094413"
format_stamp: "Formatted at 2026-09-23 20:09:34 on dist-test-slave-th4g"
server_key: "12f01e1f6d0d0154e92a79d1431b13c9"
server_key_iv: "76c27d6fa37b1ed0aa68d51a195bc32f"
server_key_version: "encryptionkey@0"
I20260923 20:09:34.217819 7541 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260923 20:09:34.221716 7556 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:09:34.223933 7541 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260923 20:09:34.224157 7541 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790194121577027-1706-0/minicluster-data/ts-1/data,/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790194121577027-1706-0/minicluster-data/ts-1/wal
uuid: "b2f17c13664541b6abee0b077b094413"
format_stamp: "Formatted at 2026-09-23 20:09:34 on dist-test-slave-th4g"
server_key: "12f01e1f6d0d0154e92a79d1431b13c9"
server_key_iv: "76c27d6fa37b1ed0aa68d51a195bc32f"
server_key_version: "encryptionkey@0"
I20260923 20:09:34.224408 7541 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790194121577027-1706-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790194121577027-1706-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790194121577027-1706-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:09:34.264101 7541 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:09:34.268728 7541 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:09:34.269081 7541 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:09:34.270339 7541 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 20:09:34.271896 7541 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 20:09:34.272012 7541 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 20:09:34.272164 7541 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 20:09:34.272217 7541 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 20:09:34.311542 7541 rpc_server.cc:307] RPC server started. Bound to: 127.1.170.130:45947
I20260923 20:09:34.311580 7668 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.170.130:45947 every 8 connection(s)
I20260923 20:09:34.312995 7541 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790194121577027-1706-0/minicluster-data/ts-1/data/info.pb
I20260923 20:09:34.318940 1706 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqUuku6/build/asan/bin/kudu as pid 7541
I20260923 20:09:34.319240 1706 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790194121577027-1706-0/minicluster-data/ts-1/wal/instance
I20260923 20:09:34.319937 1706 external_mini_cluster.cc:1428] Setting key 38da343547272b7ec30053fb693139e3
Sep 23 20:09:34 dist-test-slave-th4g krb5kdc[7301](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194174, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.170.130@KRBTEST.COM for kudu/127.1.170.190@KRBTEST.COM
I20260923 20:09:34.342418 7671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:09:34.315778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.170.130:53467 (local address 127.1.170.190:43197)
0923 20:09:34.316354 (+ 576us) server_negotiation.cc:206] Beginning negotiation
0923 20:09:34.316360 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:09:34.317714 (+ 1354us) server_negotiation.cc:407] Connection header received
0923 20:09:34.321383 (+ 3669us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:09:34.321389 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:09:34.321491 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:09:34.321677 (+ 186us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:09:34.324941 (+ 3264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:09:34.326217 (+ 1276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:09:34.328436 (+ 2219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:09:34.329281 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:09:34.329434 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:09:34.334323 (+ 4889us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:09:34.334331 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:09:34.334335 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:09:34.334387 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:09:34.338389 (+ 4002us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:09:34.339626 (+ 1237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:09:34.339631 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:09:34.339634 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:09:34.339789 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:09:34.340384 (+ 595us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:09:34.340388 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:09:34.340390 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:09:34.340696 (+ 306us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:09:34.340821 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0923 20:09:34.341863 (+ 1042us) server_negotiation.cc:299] Negotiation successful
0923 20:09:34.342138 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":445,"thread_start_us":325,"threads_started":1}
WARNING: no policy specified for kudu/127.1.170.131@KRBTEST.COM; defaulting to no policy
I20260923 20:09:34.344053 7672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:09:34.316444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.190:43197 (local address 127.1.170.130:53467)
0923 20:09:34.317424 (+ 980us) negotiation.cc:107] Waiting for socket to connect
0923 20:09:34.317493 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0923 20:09:34.320952 (+ 3459us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:09:34.322260 (+ 1308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:09:34.322273 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:09:34.323240 (+ 967us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:09:34.324604 (+ 1364us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:09:34.324654 (+ 50us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:09:34.326466 (+ 1812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:09:34.326472 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:09:34.328229 (+ 1757us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:09:34.328252 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:09:34.329696 (+ 1444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:09:34.329703 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:09:34.330022 (+ 319us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:09:34.331103 (+ 1081us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:09:34.331128 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:09:34.333857 (+ 2729us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:09:34.338585 (+ 4728us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:09:34.338600 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:09:34.338655 (+ 55us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:09:34.339383 (+ 728us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:09:34.339921 (+ 538us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:09:34.339929 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:09:34.339936 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:09:34.340227 (+ 291us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:09:34.340845 (+ 618us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:09:34.340856 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:09:34.341391 (+ 535us) client_negotiation.cc:769] Sending connection context
0923 20:09:34.341740 (+ 349us) client_negotiation.cc:240] Negotiation successful
0923 20:09:34.342391 (+ 651us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":530,"thread_start_us":320,"threads_started":1}
I20260923 20:09:34.345758 7669 heartbeater.cc:347] Connected to a master server at 127.1.170.190:43197
I20260923 20:09:34.346167 7669 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.170.131@KRBTEST.COM" created.
I20260923 20:09:34.347088 7669 heartbeater.cc:511] Master 127.1.170.190:43197 requested a full tablet report, sending...
I20260923 20:09:34.348714 7346 ts_manager.cc:194] Registered new tserver with Master: b2f17c13664541b6abee0b077b094413 (127.1.170.130:45947)
I20260923 20:09:34.350025 7346 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.170.130@KRBTEST.COM'} at 127.1.170.130:53467
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.170.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790194121577027-1706-0/krb5kdc/kudu_127.1.170.131.keytab.
Entry for principal kudu/127.1.170.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790194121577027-1706-0/krb5kdc/kudu_127.1.170.131.keytab.
I20260923 20:09:34.376175 1706 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqUuku6/build/asan/bin/kudu
/tmp/dist-test-taskqUuku6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790194121577027-1706-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790194121577027-1706-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790194121577027-1706-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790194121577027-1706-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.170.131:0
--local_ip_for_outbound_sockets=127.1.170.131
--webserver_interface=127.1.170.131
--webserver_port=0
--tserver_master_addrs=127.1.170.190:43197
--keytab_file=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790194121577027-1706-0/krb5kdc/kudu_127.1.170.131.keytab
--principal=kudu/127.1.170.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.170.148:43289
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790194121577027-1706-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790194121577027-1706-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790194121577027-1706-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:09:34.708218 7675 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:09:34.708506 7675 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:09:34.708600 7675 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:09:34.715813 7675 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:09:34.715931 7675 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:09:34.716041 7675 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.170.131
I20260923 20:09:34.726511 7675 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.170.148:43289
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790194121577027-1706-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790194121577027-1706-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790194121577027-1706-0/krb5kdc/kudu_127.1.170.131.keytab
--principal=kudu/127.1.170.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.170.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790194121577027-1706-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.170.131
--webserver_port=0
--tserver_master_addrs=127.1.170.190:43197
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.170.131
--log_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790194121577027-1706-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:09:34.727684 7675 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:09:34.729085 7675 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:09:34.742623 7680 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:09:34.742663 7681 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:09:34.743366 7683 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:09:34.745461 7675 server_base.cc:1034] running on GCE node
I20260923 20:09:34.746196 7675 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:09:34.747216 7675 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:09:34.748448 7675 hybrid_clock.cc:648] HybridClock initialized: now 1790194174748407 us; error 28 us; skew 500 ppm
I20260923 20:09:34.748799 7675 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:09:34 dist-test-slave-th4g krb5kdc[7301](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194174, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.170.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:09:34.751920 7675 init.cc:377] Logged in from keytab as kudu/127.1.170.131@KRBTEST.COM (short username kudu)
I20260923 20:09:34.753401 7675 webserver.cc:460] Webserver started at http://127.1.170.131:43891/ using document root <none> and password file <none>
I20260923 20:09:34.754168 7675 fs_manager.cc:359] Metadata directory not provided
I20260923 20:09:34.754303 7675 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:09:34.754598 7675 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:09:34.757455 7675 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790194121577027-1706-0/minicluster-data/ts-2/data/instance:
uuid: "279021fc434b473c92ae664d8b3b7945"
format_stamp: "Formatted at 2026-09-23 20:09:34 on dist-test-slave-th4g"
server_key: "ef1d84e8aedb746eca093b7b77368911"
server_key_iv: "92658634a0ffe8ba240dee9dacc7efb4"
server_key_version: "encryptionkey@0"
I20260923 20:09:34.758262 7675 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790194121577027-1706-0/minicluster-data/ts-2/wal/instance:
uuid: "279021fc434b473c92ae664d8b3b7945"
format_stamp: "Formatted at 2026-09-23 20:09:34 on dist-test-slave-th4g"
server_key: "ef1d84e8aedb746eca093b7b77368911"
server_key_iv: "92658634a0ffe8ba240dee9dacc7efb4"
server_key_version: "encryptionkey@0"
I20260923 20:09:34.763315 7675 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260923 20:09:34.766729 7690 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:09:34.768162 7675 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260923 20:09:34.768409 7675 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790194121577027-1706-0/minicluster-data/ts-2/data,/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790194121577027-1706-0/minicluster-data/ts-2/wal
uuid: "279021fc434b473c92ae664d8b3b7945"
format_stamp: "Formatted at 2026-09-23 20:09:34 on dist-test-slave-th4g"
server_key: "ef1d84e8aedb746eca093b7b77368911"
server_key_iv: "92658634a0ffe8ba240dee9dacc7efb4"
server_key_version: "encryptionkey@0"
I20260923 20:09:34.768626 7675 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790194121577027-1706-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790194121577027-1706-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790194121577027-1706-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:09:34.780676 7535 heartbeater.cc:503] Master 127.1.170.190:43197 was elected leader, sending a full tablet report...
I20260923 20:09:34.787669 7675 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:09:34.791864 7675 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:09:34.792107 7675 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:09:34.793010 7675 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 20:09:34.794425 7675 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 20:09:34.794497 7675 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 20:09:34.794582 7675 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 20:09:34.794631 7675 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 20:09:34.827718 7675 rpc_server.cc:307] RPC server started. Bound to: 127.1.170.131:43069
I20260923 20:09:34.827759 7802 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.170.131:43069 every 8 connection(s)
I20260923 20:09:34.829028 7675 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790194121577027-1706-0/minicluster-data/ts-2/data/info.pb
I20260923 20:09:34.837986 1706 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqUuku6/build/asan/bin/kudu as pid 7675
I20260923 20:09:34.838246 1706 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790194121577027-1706-0/minicluster-data/ts-2/wal/instance
I20260923 20:09:34.838686 1706 external_mini_cluster.cc:1428] Setting key c537aec284f15e44e02311515d1ca33b
Sep 23 20:09:34 dist-test-slave-th4g krb5kdc[7301](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194174, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.170.131@KRBTEST.COM for kudu/127.1.170.190@KRBTEST.COM
I20260923 20:09:34.849123 7671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:09:34.831697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.170.131:46503 (local address 127.1.170.190:43197)
0923 20:09:34.831873 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0923 20:09:34.831878 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 20:09:34.833051 (+ 1173us) server_negotiation.cc:407] Connection header received
0923 20:09:34.834953 (+ 1902us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:09:34.834959 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:09:34.835066 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:09:34.835179 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:09:34.836651 (+ 1472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:09:34.837438 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:09:34.839462 (+ 2024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:09:34.840057 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:09:34.840181 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:09:34.843719 (+ 3538us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:09:34.843725 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:09:34.843730 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:09:34.843773 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:09:34.846252 (+ 2479us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:09:34.846950 (+ 698us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:09:34.846956 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:09:34.846959 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:09:34.847075 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:09:34.847491 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:09:34.847496 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:09:34.847499 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:09:34.847763 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:09:34.847918 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0923 20:09:34.848555 (+ 637us) server_negotiation.cc:299] Negotiation successful
0923 20:09:34.848785 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260923 20:09:34.850106 7805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:09:34.832196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.190:43197 (local address 127.1.170.131:46503)
0923 20:09:34.832894 (+ 698us) negotiation.cc:107] Waiting for socket to connect
0923 20:09:34.832939 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0923 20:09:34.834670 (+ 1731us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:09:34.835405 (+ 735us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:09:34.835414 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:09:34.835924 (+ 510us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:09:34.836488 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:09:34.836504 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:09:34.837595 (+ 1091us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:09:34.837600 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:09:34.839080 (+ 1480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:09:34.839092 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:09:34.840278 (+ 1186us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:09:34.840292 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:09:34.840561 (+ 269us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:09:34.841346 (+ 785us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:09:34.841385 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:09:34.843517 (+ 2132us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:09:34.846391 (+ 2874us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:09:34.846400 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:09:34.846419 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:09:34.846824 (+ 405us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:09:34.847190 (+ 366us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:09:34.847196 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:09:34.847200 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:09:34.847376 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:09:34.847885 (+ 509us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:09:34.847893 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:09:34.848216 (+ 323us) client_negotiation.cc:769] Sending connection context
0923 20:09:34.848467 (+ 251us) client_negotiation.cc:240] Negotiation successful
0923 20:09:34.848995 (+ 528us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":474,"thread_start_us":279,"threads_started":1}
I20260923 20:09:34.851423 7803 heartbeater.cc:347] Connected to a master server at 127.1.170.190:43197
I20260923 20:09:34.851742 7803 heartbeater.cc:464] Registering TS with master...
I20260923 20:09:34.852676 7803 heartbeater.cc:511] Master 127.1.170.190:43197 requested a full tablet report, sending...
I20260923 20:09:34.854426 7346 ts_manager.cc:194] Registered new tserver with Master: 279021fc434b473c92ae664d8b3b7945 (127.1.170.131:43069)
I20260923 20:09:34.855247 7346 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.170.131@KRBTEST.COM'} at 127.1.170.131:46503
I20260923 20:09:34.866641 1706 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790194121577027-1706-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790194121577027-1706-0/krb5kdc/test-admin.keytab.
Sep 23 20:09:34 dist-test-slave-th4g krb5kdc[7301](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194174, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:09:34.893388 1706 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260923 20:09:34.898358 7810 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 20:09:34.895984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52104 (local address 127.1.170.129:40137)
0923 20:09:34.896624 (+ 640us) server_negotiation.cc:206] Beginning negotiation
0923 20:09:34.896631 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:09:34.896657 (+ 26us) server_negotiation.cc:407] Connection header received
0923 20:09:34.896716 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:09:34.896722 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:09:34.896940 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:09:34.897071 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:09:34.897862 (+ 791us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52104: BlockingRecv error: recv got EOF from 127.0.0.1:52104 (error 108)
Metrics: {"server-negotiator.queue_time_us":494,"thread_start_us":355,"threads_started":1}
W20260923 20:09:34.898608 1706 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790194121577027-1706-0/krb5kdc/krb5cc' of type FILE
Sep 23 20:09:34 dist-test-slave-th4g krb5kdc[7301](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194174, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:09:34.900158 1706 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 23 20:09:34 dist-test-slave-th4g krb5kdc[7301](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194174, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:09:34.901455 1706 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 23 20:09:34 dist-test-slave-th4g krb5kdc[7301](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194174, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.170.129@KRBTEST.COM
I20260923 20:09:34.916353 7810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:09:34.903165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52118 (local address 127.1.170.129:40137)
0923 20:09:34.903374 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0923 20:09:34.903378 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 20:09:34.903489 (+ 111us) server_negotiation.cc:407] Connection header received
0923 20:09:34.903625 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:09:34.903628 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:09:34.903683 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:09:34.903760 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:09:34.904616 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:09:34.905565 (+ 949us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:09:34.906769 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:09:34.907341 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:09:34.907441 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:09:34.910896 (+ 3455us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:09:34.910907 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:09:34.910922 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:09:34.910964 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:09:34.913793 (+ 2829us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:09:34.914440 (+ 647us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:09:34.914446 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:09:34.914449 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:09:34.914557 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:09:34.914869 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:09:34.914872 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:09:34.914874 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:09:34.915238 (+ 364us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:09:34.915478 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0923 20:09:34.915785 (+ 307us) server_negotiation.cc:299] Negotiation successful
0923 20:09:34.916048 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
W20260923 20:09:34.922509 7810 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 20:09:34.920537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52130 (local address 127.1.170.129:40137)
0923 20:09:34.920733 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0923 20:09:34.920739 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:09:34.921004 (+ 265us) server_negotiation.cc:407] Connection header received
0923 20:09:34.921256 (+ 252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:09:34.921266 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:09:34.921335 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:09:34.921433 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:09:34.922248 (+ 815us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52130: BlockingRecv error: recv got EOF from 127.0.0.1:52130 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
W20260923 20:09:34.922672 1706 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790194121577027-1706-0/krb5kdc/krb5cc' of type FILE
Sep 23 20:09:34 dist-test-slave-th4g krb5kdc[7301](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194174, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:09:34.924077 1706 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 23 20:09:34 dist-test-slave-th4g krb5kdc[7301](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194174, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:09:34.925339 1706 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 23 20:09:34 dist-test-slave-th4g krb5kdc[7301](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194174, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.170.129@KRBTEST.COM
I20260923 20:09:34.939657 7810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:09:34.926614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52144 (local address 127.1.170.129:40137)
0923 20:09:34.926806 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0923 20:09:34.926814 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 20:09:34.927091 (+ 277us) server_negotiation.cc:407] Connection header received
0923 20:09:34.927263 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:09:34.927293 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:09:34.927348 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:09:34.927449 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:09:34.928305 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:09:34.929018 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:09:34.930286 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:09:34.930722 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:09:34.930862 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:09:34.934393 (+ 3531us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:09:34.934398 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:09:34.934401 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:09:34.934434 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:09:34.937282 (+ 2848us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:09:34.937985 (+ 703us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:09:34.937989 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:09:34.937991 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:09:34.938118 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:09:34.938443 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:09:34.938446 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:09:34.938448 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:09:34.938733 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:09:34.938916 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0923 20:09:34.939151 (+ 235us) server_negotiation.cc:299] Negotiation successful
0923 20:09:34.939372 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
W20260923 20:09:34.944029 7810 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 20:09:34.942282 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52156 (local address 127.1.170.129:40137)
0923 20:09:34.942485 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0923 20:09:34.942494 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 20:09:34.942766 (+ 272us) server_negotiation.cc:407] Connection header received
0923 20:09:34.942955 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:09:34.942960 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:09:34.943036 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:09:34.943137 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:09:34.943807 (+ 670us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52156: BlockingRecv error: recv got EOF from 127.0.0.1:52156 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
W20260923 20:09:34.944301 1706 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790194121577027-1706-0/krb5kdc/krb5cc' of type FILE
Sep 23 20:09:34 dist-test-slave-th4g krb5kdc[7301](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194174, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:09:34.945763 1706 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 23 20:09:34 dist-test-slave-th4g krb5kdc[7301](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194174, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:09:34.947130 1706 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 23 20:09:34 dist-test-slave-th4g krb5kdc[7301](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194174, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.170.129@KRBTEST.COM
I20260923 20:09:34.962025 7810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:09:34.948586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52158 (local address 127.1.170.129:40137)
0923 20:09:34.948785 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0923 20:09:34.948791 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:09:34.948871 (+ 80us) server_negotiation.cc:407] Connection header received
0923 20:09:34.948988 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:09:34.948992 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:09:34.949048 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:09:34.949130 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:09:34.950171 (+ 1041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:09:34.950885 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:09:34.952353 (+ 1468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:09:34.952759 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:09:34.952874 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:09:34.956854 (+ 3980us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:09:34.956859 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:09:34.956863 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:09:34.956897 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:09:34.959520 (+ 2623us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:09:34.960201 (+ 681us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:09:34.960204 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:09:34.960207 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:09:34.960381 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:09:34.960708 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:09:34.960711 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:09:34.960713 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:09:34.960986 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:09:34.961200 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0923 20:09:34.961440 (+ 240us) server_negotiation.cc:299] Negotiation successful
0923 20:09:34.961633 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260923 20:09:34.963673 1706 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskqUuku6/build/asan/bin/kudu pid 7407
I20260923 20:09:34.975064 7810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:09:34.964387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52168 (local address 127.1.170.129:40137)
0923 20:09:34.964531 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0923 20:09:34.964535 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 20:09:34.964863 (+ 328us) server_negotiation.cc:407] Connection header received
0923 20:09:34.964982 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:09:34.964986 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:09:34.965036 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:09:34.965110 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:09:34.966037 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:09:34.966675 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:09:34.967835 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:09:34.968237 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:09:34.968336 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:09:34.970334 (+ 1998us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:09:34.970339 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:09:34.970342 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:09:34.970369 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:09:34.972668 (+ 2299us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:09:34.973338 (+ 670us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:09:34.973342 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:09:34.973345 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:09:34.973480 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:09:34.973797 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:09:34.973800 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:09:34.973802 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:09:34.974132 (+ 330us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:09:34.974322 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0923 20:09:34.974630 (+ 308us) server_negotiation.cc:299] Negotiation successful
0923 20:09:34.974810 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260923 20:09:34.975637 7530 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:09:35.060698 1706 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqUuku6/build/asan/bin/kudu with pid 7407
I20260923 20:09:35.077054 1706 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskqUuku6/build/asan/bin/kudu pid 7541
Sep 23 20:09:35 dist-test-slave-th4g krb5kdc[7301](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194174, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.170.130@KRBTEST.COM
I20260923 20:09:35.094339 7823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:09:35.078096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49432 (local address 127.1.170.130:45947)
0923 20:09:35.078668 (+ 572us) server_negotiation.cc:206] Beginning negotiation
0923 20:09:35.078675 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:09:35.078698 (+ 23us) server_negotiation.cc:407] Connection header received
0923 20:09:35.078757 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:09:35.078763 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:09:35.078963 (+ 200us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:09:35.079087 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:09:35.080346 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:09:35.081331 (+ 985us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:09:35.083294 (+ 1963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:09:35.083836 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:09:35.083940 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:09:35.088489 (+ 4549us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:09:35.088506 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:09:35.088527 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:09:35.088582 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:09:35.091625 (+ 3043us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:09:35.092389 (+ 764us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:09:35.092397 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:09:35.092399 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:09:35.092524 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:09:35.092843 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:09:35.092846 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:09:35.092848 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:09:35.093181 (+ 333us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:09:35.093395 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0923 20:09:35.093668 (+ 273us) server_negotiation.cc:299] Negotiation successful
0923 20:09:35.093945 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":442,"thread_start_us":261,"threads_started":1}
I20260923 20:09:35.095561 7664 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:09:35.177110 1706 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqUuku6/build/asan/bin/kudu with pid 7541
I20260923 20:09:35.192562 1706 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskqUuku6/build/asan/bin/kudu pid 7675
Sep 23 20:09:35 dist-test-slave-th4g krb5kdc[7301](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194174, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.170.131@KRBTEST.COM
I20260923 20:09:35.207458 7825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:09:35.193552 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34660 (local address 127.1.170.131:43069)
0923 20:09:35.194216 (+ 664us) server_negotiation.cc:206] Beginning negotiation
0923 20:09:35.194223 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:09:35.194251 (+ 28us) server_negotiation.cc:407] Connection header received
0923 20:09:35.194310 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:09:35.194316 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:09:35.194509 (+ 193us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:09:35.194636 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:09:35.195595 (+ 959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:09:35.196583 (+ 988us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:09:35.197842 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:09:35.198502 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:09:35.198602 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:09:35.202270 (+ 3668us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:09:35.202281 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:09:35.202293 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:09:35.202329 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:09:35.205047 (+ 2718us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:09:35.205693 (+ 646us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:09:35.205700 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:09:35.205702 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:09:35.205828 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:09:35.206225 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:09:35.206228 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:09:35.206230 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:09:35.206549 (+ 319us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:09:35.206763 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0923 20:09:35.206995 (+ 232us) server_negotiation.cc:299] Negotiation successful
0923 20:09:35.207178 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":532,"thread_start_us":391,"threads_started":1}
I20260923 20:09:35.208685 7798 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:09:35.299059 1706 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqUuku6/build/asan/bin/kudu with pid 7675
I20260923 20:09:35.314083 1706 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskqUuku6/build/asan/bin/kudu pid 7314
I20260923 20:09:35.315414 7376 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:09:35.418784 1706 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqUuku6/build/asan/bin/kudu with pid 7314
2026-09-23T20:09:35Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (4088 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194121577027-1706-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 20:09:35 dist-test-slave-th4g krb5kdc[7830](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 20:09:35 dist-test-slave-th4g krb5kdc[7830](info): set up 2 sockets
Sep 23 20:09:35 dist-test-slave-th4g krb5kdc[7830](info): commencing operation
krb5kdc: starting...
W20260923 20:09:36.518886 1706 mini_kdc.cc:121] Time spent starting KDC: real 1.057s user 0.002s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194121577027-1706-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194121577027-1706-0/krb5kdc/test-user.keytab.
Sep 23 20:09:36 dist-test-slave-th4g krb5kdc[7830](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194176, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T20:09:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T20:09:36Z Disabled control of system clock
WARNING: no policy specified for oryx/127.1.170.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.1.170.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.1.170.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194121577027-1706-0/krb5kdc/oryx_127.1.170.190.keytab.
Entry for principal oryx/127.1.170.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194121577027-1706-0/krb5kdc/oryx_127.1.170.190.keytab.
I20260923 20:09:36.735082 1706 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqUuku6/build/asan/bin/kudu
/tmp/dist-test-taskqUuku6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194121577027-1706-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194121577027-1706-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194121577027-1706-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194121577027-1706-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.170.190:42111
--webserver_interface=127.1.170.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194121577027-1706-0/krb5kdc/oryx_127.1.170.190.keytab
--principal=oryx/127.1.170.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.170.148:44293
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.170.190:42111
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194121577027-1706-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194121577027-1706-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194121577027-1706-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:09:37.061770 7843 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:09:37.062182 7843 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:09:37.062361 7843 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:09:37.070627 7843 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 20:09:37.070820 7843 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:09:37.070932 7843 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:09:37.071038 7843 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 20:09:37.071105 7843 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 20:09:37.081274 7843 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.170.148:44293
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194121577027-1706-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194121577027-1706-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.170.190:42111
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194121577027-1706-0/krb5kdc/oryx_127.1.170.190.keytab
--principal=oryx/127.1.170.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.170.190:42111
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194121577027-1706-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.170.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194121577027-1706-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:09:37.082763 7843 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:09:37.084156 7843 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:09:37.096894 7849 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:09:37.096913 7848 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:09:37.096839 7851 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:09:37.099542 7843 server_base.cc:1034] running on GCE node
I20260923 20:09:37.100437 7843 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:09:37.102037 7843 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:09:37.103294 7843 hybrid_clock.cc:648] HybridClock initialized: now 1790194177103259 us; error 76 us; skew 500 ppm
I20260923 20:09:37.103694 7843 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:09:37 dist-test-slave-th4g krb5kdc[7830](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194177, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.170.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:09:37.113552 7843 init.cc:377] Logged in from keytab as oryx/127.1.170.190@KRBTEST.COM (short username oryx)
I20260923 20:09:37.115412 7843 webserver.cc:460] Webserver started at http://127.1.170.190:34031/ using document root <none> and password file <none>
I20260923 20:09:37.116191 7843 fs_manager.cc:359] Metadata directory not provided
I20260923 20:09:37.116339 7843 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:09:37.116650 7843 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:09:37.119817 7843 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194121577027-1706-0/minicluster-data/master-0/data/instance:
uuid: "f83afa4c3fa644f08196bf2ef04d733b"
format_stamp: "Formatted at 2026-09-23 20:09:37 on dist-test-slave-th4g"
server_key: "0a74d937141087fe37bbcf5aad1c188f"
server_key_iv: "464839076d83d23419d5368218c77e3e"
server_key_version: "encryptionkey@0"
I20260923 20:09:37.120599 7843 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194121577027-1706-0/minicluster-data/master-0/wal/instance:
uuid: "f83afa4c3fa644f08196bf2ef04d733b"
format_stamp: "Formatted at 2026-09-23 20:09:37 on dist-test-slave-th4g"
server_key: "0a74d937141087fe37bbcf5aad1c188f"
server_key_iv: "464839076d83d23419d5368218c77e3e"
server_key_version: "encryptionkey@0"
I20260923 20:09:37.126230 7843 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.004s
I20260923 20:09:37.129997 7858 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:09:37.131817 7843 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260923 20:09:37.132019 7843 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194121577027-1706-0/minicluster-data/master-0/data,/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194121577027-1706-0/minicluster-data/master-0/wal
uuid: "f83afa4c3fa644f08196bf2ef04d733b"
format_stamp: "Formatted at 2026-09-23 20:09:37 on dist-test-slave-th4g"
server_key: "0a74d937141087fe37bbcf5aad1c188f"
server_key_iv: "464839076d83d23419d5368218c77e3e"
server_key_version: "encryptionkey@0"
I20260923 20:09:37.132230 7843 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194121577027-1706-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194121577027-1706-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194121577027-1706-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:09:37.164953 7843 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:09:37.169632 7843 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:09:37.170077 7843 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:09:37.189674 7843 rpc_server.cc:307] RPC server started. Bound to: 127.1.170.190:42111
I20260923 20:09:37.189693 7909 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.170.190:42111 every 8 connection(s)
I20260923 20:09:37.191138 7843 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194121577027-1706-0/minicluster-data/master-0/data/info.pb
I20260923 20:09:37.194944 7910 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:09:37.197120 1706 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqUuku6/build/asan/bin/kudu as pid 7843
I20260923 20:09:37.197388 1706 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194121577027-1706-0/minicluster-data/master-0/wal/instance
I20260923 20:09:37.198176 1706 external_mini_cluster.cc:1428] Setting key 205ef31d3e3aadd41d91e570873632a5
I20260923 20:09:37.208343 7910 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f83afa4c3fa644f08196bf2ef04d733b: Bootstrap starting.
I20260923 20:09:37.212374 7910 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f83afa4c3fa644f08196bf2ef04d733b: Neither blocks nor log segments found. Creating new log.
Sep 23 20:09:37 dist-test-slave-th4g krb5kdc[7830](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194176, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.1.170.190@KRBTEST.COM
I20260923 20:09:37.213744 7910 log.cc:824] T 00000000000000000000000000000000 P f83afa4c3fa644f08196bf2ef04d733b: Log is configured to *not* fsync() on all Append() calls
I20260923 20:09:37.217278 7910 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f83afa4c3fa644f08196bf2ef04d733b: No bootstrap required, opened a new log
I20260923 20:09:37.222923 7913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:09:37.200451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41822 (local address 127.1.170.190:42111)
0923 20:09:37.201318 (+ 867us) server_negotiation.cc:206] Beginning negotiation
0923 20:09:37.201329 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0923 20:09:37.201373 (+ 44us) server_negotiation.cc:407] Connection header received
0923 20:09:37.203068 (+ 1695us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:09:37.203090 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:09:37.203477 (+ 387us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:09:37.203823 (+ 346us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:09:37.205579 (+ 1756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:09:37.207047 (+ 1468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:09:37.208744 (+ 1697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:09:37.209458 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:09:37.209597 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:09:37.214767 (+ 5170us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:09:37.214779 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:09:37.214801 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:09:37.214847 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:09:37.218295 (+ 3448us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:09:37.219260 (+ 965us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:09:37.219267 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:09:37.219274 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:09:37.219459 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:09:37.219889 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:09:37.219893 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:09:37.219895 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:09:37.220507 (+ 612us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:09:37.220744 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0923 20:09:37.221302 (+ 558us) server_negotiation.cc:299] Negotiation successful
0923 20:09:37.221759 (+ 457us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":660,"thread_start_us":386,"threads_started":1}
I20260923 20:09:37.227224 7910 raft_consensus.cc:348] T 00000000000000000000000000000000 P f83afa4c3fa644f08196bf2ef04d733b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f83afa4c3fa644f08196bf2ef04d733b" member_type: VOTER last_known_addr { host: "127.1.170.190" port: 42111 } }
I20260923 20:09:37.227639 7910 raft_consensus.cc:374] T 00000000000000000000000000000000 P f83afa4c3fa644f08196bf2ef04d733b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:09:37.227742 7910 raft_consensus.cc:729] T 00000000000000000000000000000000 P f83afa4c3fa644f08196bf2ef04d733b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f83afa4c3fa644f08196bf2ef04d733b, State: Initialized, Role: FOLLOWER
I20260923 20:09:37.228332 7910 consensus_queue.cc:260] T 00000000000000000000000000000000 P f83afa4c3fa644f08196bf2ef04d733b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f83afa4c3fa644f08196bf2ef04d733b" member_type: VOTER last_known_addr { host: "127.1.170.190" port: 42111 } }
I20260923 20:09:37.228536 7910 raft_consensus.cc:388] T 00000000000000000000000000000000 P f83afa4c3fa644f08196bf2ef04d733b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 20:09:37.228708 7910 raft_consensus.cc:482] T 00000000000000000000000000000000 P f83afa4c3fa644f08196bf2ef04d733b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 20:09:37.228860 7910 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f83afa4c3fa644f08196bf2ef04d733b [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:09:37.231293 7910 raft_consensus.cc:504] T 00000000000000000000000000000000 P f83afa4c3fa644f08196bf2ef04d733b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f83afa4c3fa644f08196bf2ef04d733b" member_type: VOTER last_known_addr { host: "127.1.170.190" port: 42111 } }
I20260923 20:09:37.231822 7910 leader_election.cc:302] T 00000000000000000000000000000000 P f83afa4c3fa644f08196bf2ef04d733b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f83afa4c3fa644f08196bf2ef04d733b; no voters:
I20260923 20:09:37.232451 7910 leader_election.cc:288] T 00000000000000000000000000000000 P f83afa4c3fa644f08196bf2ef04d733b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 20:09:37.232600 7915 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f83afa4c3fa644f08196bf2ef04d733b [term 1 FOLLOWER]: Leader election won for term 1
I20260923 20:09:37.233319 7915 raft_consensus.cc:686] T 00000000000000000000000000000000 P f83afa4c3fa644f08196bf2ef04d733b [term 1 LEADER]: Becoming Leader. State: Replica: f83afa4c3fa644f08196bf2ef04d733b, State: Running, Role: LEADER
I20260923 20:09:37.233923 7915 consensus_queue.cc:237] T 00000000000000000000000000000000 P f83afa4c3fa644f08196bf2ef04d733b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f83afa4c3fa644f08196bf2ef04d733b" member_type: VOTER last_known_addr { host: "127.1.170.190" port: 42111 } }
I20260923 20:09:37.234730 7910 sys_catalog.cc:564] T 00000000000000000000000000000000 P f83afa4c3fa644f08196bf2ef04d733b [sys.catalog]: configured and running, proceeding with master startup.
I20260923 20:09:37.236714 7917 sys_catalog.cc:455] T 00000000000000000000000000000000 P f83afa4c3fa644f08196bf2ef04d733b [sys.catalog]: SysCatalogTable state changed. Reason: New leader f83afa4c3fa644f08196bf2ef04d733b. Latest consensus state: current_term: 1 leader_uuid: "f83afa4c3fa644f08196bf2ef04d733b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f83afa4c3fa644f08196bf2ef04d733b" member_type: VOTER last_known_addr { host: "127.1.170.190" port: 42111 } } }
I20260923 20:09:37.237227 7917 sys_catalog.cc:458] T 00000000000000000000000000000000 P f83afa4c3fa644f08196bf2ef04d733b [sys.catalog]: This master's current role is: LEADER
I20260923 20:09:37.237869 7916 sys_catalog.cc:455] T 00000000000000000000000000000000 P f83afa4c3fa644f08196bf2ef04d733b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f83afa4c3fa644f08196bf2ef04d733b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f83afa4c3fa644f08196bf2ef04d733b" member_type: VOTER last_known_addr { host: "127.1.170.190" port: 42111 } } }
I20260923 20:09:37.238269 7916 sys_catalog.cc:458] T 00000000000000000000000000000000 P f83afa4c3fa644f08196bf2ef04d733b [sys.catalog]: This master's current role is: LEADER
I20260923 20:09:37.240499 7921 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 20:09:37.246241 7921 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 20:09:37.256518 7921 catalog_manager.cc:1347] Generated new cluster ID: 62f93adc2448407bb12c561910d92d81
I20260923 20:09:37.256676 7921 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 20:09:37.280790 7921 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 20:09:37.282243 7921 catalog_manager.cc:1504] Loading token signing keys...
I20260923 20:09:37.301745 7921 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f83afa4c3fa644f08196bf2ef04d733b: Generated new TSK 0
I20260923 20:09:37.302953 7921 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.1.170.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.1.170.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.1.170.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194121577027-1706-0/krb5kdc/oryx_127.1.170.129.keytab.
Entry for principal oryx/127.1.170.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194121577027-1706-0/krb5kdc/oryx_127.1.170.129.keytab.
I20260923 20:09:37.370733 1706 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqUuku6/build/asan/bin/kudu
/tmp/dist-test-taskqUuku6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194121577027-1706-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194121577027-1706-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194121577027-1706-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194121577027-1706-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.170.129:0
--local_ip_for_outbound_sockets=127.1.170.129
--webserver_interface=127.1.170.129
--webserver_port=0
--tserver_master_addrs=127.1.170.190:42111
--keytab_file=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194121577027-1706-0/krb5kdc/oryx_127.1.170.129.keytab
--principal=oryx/127.1.170.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.170.148:44293
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194121577027-1706-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194121577027-1706-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194121577027-1706-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:09:37.716755 7936 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:09:37.717034 7936 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:09:37.717127 7936 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:09:37.724768 7936 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:09:37.724889 7936 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:09:37.725000 7936 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.170.129
I20260923 20:09:37.735873 7936 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.170.148:44293
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194121577027-1706-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194121577027-1706-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194121577027-1706-0/krb5kdc/oryx_127.1.170.129.keytab
--principal=oryx/127.1.170.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.170.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194121577027-1706-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.170.129
--webserver_port=0
--tserver_master_addrs=127.1.170.190:42111
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.170.129
--log_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194121577027-1706-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:09:37.737377 7936 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:09:37.738754 7936 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:09:37.752202 7942 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:09:37.752333 7944 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:09:37.752322 7941 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:09:37.753296 7936 server_base.cc:1034] running on GCE node
I20260923 20:09:37.753940 7936 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:09:37.754956 7936 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:09:37.756215 7936 hybrid_clock.cc:648] HybridClock initialized: now 1790194177756132 us; error 63 us; skew 500 ppm
I20260923 20:09:37.756628 7936 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:09:37 dist-test-slave-th4g krb5kdc[7830](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194177, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.170.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:09:37.760824 7936 init.cc:377] Logged in from keytab as oryx/127.1.170.129@KRBTEST.COM (short username oryx)
I20260923 20:09:37.762763 7936 webserver.cc:460] Webserver started at http://127.1.170.129:34051/ using document root <none> and password file <none>
I20260923 20:09:37.763664 7936 fs_manager.cc:359] Metadata directory not provided
I20260923 20:09:37.763827 7936 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:09:37.764207 7936 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:09:37.767409 7936 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194121577027-1706-0/minicluster-data/ts-0/data/instance:
uuid: "e120ba40339e4897a3aa5180c4c45acf"
format_stamp: "Formatted at 2026-09-23 20:09:37 on dist-test-slave-th4g"
server_key: "f5b649f1f57f06efbb53de215791c76c"
server_key_iv: "45d7cb2b88063be3fee557c7c9d404f9"
server_key_version: "encryptionkey@0"
I20260923 20:09:37.768087 7936 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194121577027-1706-0/minicluster-data/ts-0/wal/instance:
uuid: "e120ba40339e4897a3aa5180c4c45acf"
format_stamp: "Formatted at 2026-09-23 20:09:37 on dist-test-slave-th4g"
server_key: "f5b649f1f57f06efbb53de215791c76c"
server_key_iv: "45d7cb2b88063be3fee557c7c9d404f9"
server_key_version: "encryptionkey@0"
I20260923 20:09:37.773077 7936 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260923 20:09:37.776705 7951 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:09:37.778293 7936 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260923 20:09:37.778439 7936 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194121577027-1706-0/minicluster-data/ts-0/data,/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194121577027-1706-0/minicluster-data/ts-0/wal
uuid: "e120ba40339e4897a3aa5180c4c45acf"
format_stamp: "Formatted at 2026-09-23 20:09:37 on dist-test-slave-th4g"
server_key: "f5b649f1f57f06efbb53de215791c76c"
server_key_iv: "45d7cb2b88063be3fee557c7c9d404f9"
server_key_version: "encryptionkey@0"
I20260923 20:09:37.778605 7936 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194121577027-1706-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194121577027-1706-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194121577027-1706-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:09:37.792339 7936 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:09:37.796409 7936 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:09:37.796650 7936 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:09:37.797549 7936 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 20:09:37.798843 7936 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 20:09:37.798914 7936 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 20:09:37.798995 7936 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 20:09:37.799044 7936 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 20:09:37.835330 7936 rpc_server.cc:307] RPC server started. Bound to: 127.1.170.129:41229
I20260923 20:09:37.835417 8063 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.170.129:41229 every 8 connection(s)
I20260923 20:09:37.836844 7936 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194121577027-1706-0/minicluster-data/ts-0/data/info.pb
I20260923 20:09:37.843006 1706 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqUuku6/build/asan/bin/kudu as pid 7936
I20260923 20:09:37.843243 1706 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194121577027-1706-0/minicluster-data/ts-0/wal/instance
I20260923 20:09:37.843865 1706 external_mini_cluster.cc:1428] Setting key df9c63dbdf552cc59179f40b7dbbed46
Sep 23 20:09:37 dist-test-slave-th4g krb5kdc[7830](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194177, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.170.129@KRBTEST.COM for oryx/127.1.170.190@KRBTEST.COM
I20260923 20:09:37.858558 8066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:09:37.839588 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.170.129:53685 (local address 127.1.170.190:42111)
0923 20:09:37.840102 (+ 514us) server_negotiation.cc:206] Beginning negotiation
0923 20:09:37.840107 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 20:09:37.841130 (+ 1023us) server_negotiation.cc:407] Connection header received
0923 20:09:37.843080 (+ 1950us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:09:37.843085 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:09:37.843205 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:09:37.843336 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:09:37.845005 (+ 1669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:09:37.846022 (+ 1017us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:09:37.847568 (+ 1546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:09:37.848315 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:09:37.848465 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:09:37.852193 (+ 3728us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:09:37.852198 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:09:37.852202 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:09:37.852237 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:09:37.855039 (+ 2802us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:09:37.856059 (+ 1020us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:09:37.856064 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:09:37.856066 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:09:37.856199 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:09:37.856702 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:09:37.856705 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:09:37.856708 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:09:37.857003 (+ 295us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:09:37.857166 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0923 20:09:37.858100 (+ 934us) server_negotiation.cc:299] Negotiation successful
0923 20:09:37.858303 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":408,"thread_start_us":290,"threads_started":1}
I20260923 20:09:37.860126 8067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:09:37.840141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.190:42111 (local address 127.1.170.129:53685)
0923 20:09:37.840966 (+ 825us) negotiation.cc:107] Waiting for socket to connect
0923 20:09:37.841009 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0923 20:09:37.842803 (+ 1794us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:09:37.843584 (+ 781us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:09:37.843592 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:09:37.844139 (+ 547us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:09:37.844817 (+ 678us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:09:37.844862 (+ 45us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:09:37.846166 (+ 1304us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:09:37.846170 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:09:37.847404 (+ 1234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:09:37.847416 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:09:37.848530 (+ 1114us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:09:37.848534 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:09:37.848734 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:09:37.849447 (+ 713us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:09:37.849469 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:09:37.851928 (+ 2459us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:09:37.855209 (+ 3281us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:09:37.855222 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:09:37.855244 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:09:37.855916 (+ 672us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:09:37.856324 (+ 408us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:09:37.856332 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:09:37.856338 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:09:37.856581 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:09:37.857141 (+ 560us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:09:37.857152 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:09:37.857650 (+ 498us) client_negotiation.cc:769] Sending connection context
0923 20:09:37.858005 (+ 355us) client_negotiation.cc:240] Negotiation successful
0923 20:09:37.858609 (+ 604us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":560,"thread_start_us":335,"threads_started":1}
I20260923 20:09:37.861990 8064 heartbeater.cc:347] Connected to a master server at 127.1.170.190:42111
I20260923 20:09:37.862376 8064 heartbeater.cc:464] Registering TS with master...
I20260923 20:09:37.863108 8064 heartbeater.cc:511] Master 127.1.170.190:42111 requested a full tablet report, sending...
WARNING: no policy specified for oryx/127.1.170.130@KRBTEST.COM; defaulting to no policy
I20260923 20:09:37.865334 7875 ts_manager.cc:194] Registered new tserver with Master: e120ba40339e4897a3aa5180c4c45acf (127.1.170.129:41229)
I20260923 20:09:37.867015 7875 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.1.170.129@KRBTEST.COM'} at 127.1.170.129:53685
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.1.170.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.1.170.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194121577027-1706-0/krb5kdc/oryx_127.1.170.130.keytab.
Entry for principal oryx/127.1.170.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194121577027-1706-0/krb5kdc/oryx_127.1.170.130.keytab.
I20260923 20:09:37.893324 1706 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqUuku6/build/asan/bin/kudu
/tmp/dist-test-taskqUuku6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194121577027-1706-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194121577027-1706-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194121577027-1706-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194121577027-1706-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.170.130:0
--local_ip_for_outbound_sockets=127.1.170.130
--webserver_interface=127.1.170.130
--webserver_port=0
--tserver_master_addrs=127.1.170.190:42111
--keytab_file=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194121577027-1706-0/krb5kdc/oryx_127.1.170.130.keytab
--principal=oryx/127.1.170.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.170.148:44293
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194121577027-1706-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194121577027-1706-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194121577027-1706-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:09:38.218917 8070 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:09:38.219280 8070 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:09:38.219465 8070 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:09:38.226887 8070 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:09:38.227061 8070 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:09:38.227258 8070 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.170.130
I20260923 20:09:38.238750 8070 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.170.148:44293
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194121577027-1706-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194121577027-1706-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194121577027-1706-0/krb5kdc/oryx_127.1.170.130.keytab
--principal=oryx/127.1.170.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.170.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194121577027-1706-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.170.130
--webserver_port=0
--tserver_master_addrs=127.1.170.190:42111
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.170.130
--log_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194121577027-1706-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:09:38.240144 8070 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:09:38.242165 8070 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:09:38.256748 8075 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:09:38.256855 8078 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:09:38.256877 8076 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:09:38.257978 8070 server_base.cc:1034] running on GCE node
I20260923 20:09:38.258674 8070 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:09:38.259871 8070 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:09:38.261180 8070 hybrid_clock.cc:648] HybridClock initialized: now 1790194178261102 us; error 56 us; skew 500 ppm
I20260923 20:09:38.261541 8070 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:09:38 dist-test-slave-th4g krb5kdc[7830](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194178, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.170.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:09:38.265233 8070 init.cc:377] Logged in from keytab as oryx/127.1.170.130@KRBTEST.COM (short username oryx)
I20260923 20:09:38.267238 8070 webserver.cc:460] Webserver started at http://127.1.170.130:42585/ using document root <none> and password file <none>
I20260923 20:09:38.267987 8070 fs_manager.cc:359] Metadata directory not provided
I20260923 20:09:38.268121 8070 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:09:38.268452 8070 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:09:38.271613 8070 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194121577027-1706-0/minicluster-data/ts-1/data/instance:
uuid: "fee8671d00d44381ac37762a73d5ae51"
format_stamp: "Formatted at 2026-09-23 20:09:38 on dist-test-slave-th4g"
server_key: "a7d86ea9c6f6910ab02cb987366e5ded"
server_key_iv: "27ba946d1753dfd2ec27652629cad58e"
server_key_version: "encryptionkey@0"
I20260923 20:09:38.272418 8070 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194121577027-1706-0/minicluster-data/ts-1/wal/instance:
uuid: "fee8671d00d44381ac37762a73d5ae51"
format_stamp: "Formatted at 2026-09-23 20:09:38 on dist-test-slave-th4g"
server_key: "a7d86ea9c6f6910ab02cb987366e5ded"
server_key_iv: "27ba946d1753dfd2ec27652629cad58e"
server_key_version: "encryptionkey@0"
I20260923 20:09:38.278524 8070 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260923 20:09:38.283442 8085 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:09:38.285128 8070 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260923 20:09:38.285336 8070 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194121577027-1706-0/minicluster-data/ts-1/data,/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194121577027-1706-0/minicluster-data/ts-1/wal
uuid: "fee8671d00d44381ac37762a73d5ae51"
format_stamp: "Formatted at 2026-09-23 20:09:38 on dist-test-slave-th4g"
server_key: "a7d86ea9c6f6910ab02cb987366e5ded"
server_key_iv: "27ba946d1753dfd2ec27652629cad58e"
server_key_version: "encryptionkey@0"
I20260923 20:09:38.285547 8070 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194121577027-1706-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194121577027-1706-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194121577027-1706-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:09:38.305644 8070 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:09:38.310192 8070 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:09:38.310511 8070 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:09:38.311488 8070 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 20:09:38.312870 8070 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 20:09:38.312942 8070 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 20:09:38.313076 8070 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 20:09:38.313285 8070 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 20:09:38.350626 8070 rpc_server.cc:307] RPC server started. Bound to: 127.1.170.130:36095
I20260923 20:09:38.350664 8198 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.170.130:36095 every 8 connection(s)
I20260923 20:09:38.351933 8070 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194121577027-1706-0/minicluster-data/ts-1/data/info.pb
I20260923 20:09:38.354784 1706 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqUuku6/build/asan/bin/kudu as pid 8070
I20260923 20:09:38.354987 1706 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194121577027-1706-0/minicluster-data/ts-1/wal/instance
I20260923 20:09:38.355443 1706 external_mini_cluster.cc:1428] Setting key 8df24483ecdcbb209a0693ad1c4477c7
Sep 23 20:09:38 dist-test-slave-th4g krb5kdc[7830](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194178, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.170.130@KRBTEST.COM for oryx/127.1.170.190@KRBTEST.COM
I20260923 20:09:38.375099 8066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:09:38.354640 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.170.130:45739 (local address 127.1.170.190:42111)
0923 20:09:38.354864 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0923 20:09:38.354870 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:09:38.356270 (+ 1400us) server_negotiation.cc:407] Connection header received
0923 20:09:38.358193 (+ 1923us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:09:38.358201 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:09:38.358296 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:09:38.358409 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:09:38.359924 (+ 1515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:09:38.361072 (+ 1148us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:09:38.363712 (+ 2640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:09:38.364191 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:09:38.364358 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:09:38.368717 (+ 4359us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:09:38.368723 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:09:38.368726 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:09:38.368761 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:09:38.371610 (+ 2849us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:09:38.372581 (+ 971us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:09:38.372593 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:09:38.372596 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:09:38.372724 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:09:38.373188 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:09:38.373195 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:09:38.373203 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:09:38.373525 (+ 322us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:09:38.373679 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0923 20:09:38.374502 (+ 823us) server_negotiation.cc:299] Negotiation successful
0923 20:09:38.374744 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
WARNING: no policy specified for oryx/127.1.170.131@KRBTEST.COM; defaulting to no policy
I20260923 20:09:38.376151 8201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:09:38.355158 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.190:42111 (local address 127.1.170.130:45739)
0923 20:09:38.356101 (+ 943us) negotiation.cc:107] Waiting for socket to connect
0923 20:09:38.356155 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0923 20:09:38.357847 (+ 1692us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:09:38.358618 (+ 771us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:09:38.358627 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:09:38.359163 (+ 536us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:09:38.359749 (+ 586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:09:38.359766 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:09:38.361270 (+ 1504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:09:38.361279 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:09:38.363512 (+ 2233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:09:38.363531 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:09:38.364455 (+ 924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:09:38.364462 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:09:38.364709 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:09:38.365515 (+ 806us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:09:38.365557 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:09:38.368434 (+ 2877us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:09:38.371808 (+ 3374us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:09:38.371819 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:09:38.371842 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:09:38.372443 (+ 601us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:09:38.372844 (+ 401us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:09:38.372849 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:09:38.372853 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:09:38.373047 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:09:38.373668 (+ 621us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:09:38.373676 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:09:38.374134 (+ 458us) client_negotiation.cc:769] Sending connection context
0923 20:09:38.374410 (+ 276us) client_negotiation.cc:240] Negotiation successful
0923 20:09:38.374981 (+ 571us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":590,"thread_start_us":382,"threads_started":1}
I20260923 20:09:38.378023 8199 heartbeater.cc:347] Connected to a master server at 127.1.170.190:42111
I20260923 20:09:38.378418 8199 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.1.170.131@KRBTEST.COM" created.
I20260923 20:09:38.379428 8199 heartbeater.cc:511] Master 127.1.170.190:42111 requested a full tablet report, sending...
I20260923 20:09:38.381742 7875 ts_manager.cc:194] Registered new tserver with Master: fee8671d00d44381ac37762a73d5ae51 (127.1.170.130:36095)
I20260923 20:09:38.383010 7875 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.1.170.130@KRBTEST.COM'} at 127.1.170.130:45739
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.1.170.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194121577027-1706-0/krb5kdc/oryx_127.1.170.131.keytab.
Entry for principal oryx/127.1.170.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194121577027-1706-0/krb5kdc/oryx_127.1.170.131.keytab.
I20260923 20:09:38.406214 1706 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqUuku6/build/asan/bin/kudu
/tmp/dist-test-taskqUuku6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194121577027-1706-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194121577027-1706-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194121577027-1706-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194121577027-1706-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.170.131:0
--local_ip_for_outbound_sockets=127.1.170.131
--webserver_interface=127.1.170.131
--webserver_port=0
--tserver_master_addrs=127.1.170.190:42111
--keytab_file=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194121577027-1706-0/krb5kdc/oryx_127.1.170.131.keytab
--principal=oryx/127.1.170.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.170.148:44293
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194121577027-1706-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194121577027-1706-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194121577027-1706-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:09:38.737563 8204 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:09:38.737856 8204 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:09:38.738052 8204 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:09:38.745528 8204 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:09:38.745644 8204 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:09:38.745826 8204 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.170.131
I20260923 20:09:38.755780 8204 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.170.148:44293
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194121577027-1706-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194121577027-1706-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194121577027-1706-0/krb5kdc/oryx_127.1.170.131.keytab
--principal=oryx/127.1.170.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.170.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194121577027-1706-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.170.131
--webserver_port=0
--tserver_master_addrs=127.1.170.190:42111
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.170.131
--log_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194121577027-1706-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:09:38.757061 8204 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:09:38.758405 8204 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:09:38.771498 8209 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:09:38.771538 8212 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:09:38.771467 8210 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:09:38.772146 8204 server_base.cc:1034] running on GCE node
I20260923 20:09:38.772799 8204 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:09:38.773667 8204 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:09:38.774892 8204 hybrid_clock.cc:648] HybridClock initialized: now 1790194178774828 us; error 46 us; skew 500 ppm
I20260923 20:09:38.775239 8204 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:09:38 dist-test-slave-th4g krb5kdc[7830](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194178, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.170.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:09:38.778415 8204 init.cc:377] Logged in from keytab as oryx/127.1.170.131@KRBTEST.COM (short username oryx)
I20260923 20:09:38.779923 8204 webserver.cc:460] Webserver started at http://127.1.170.131:40925/ using document root <none> and password file <none>
I20260923 20:09:38.780625 8204 fs_manager.cc:359] Metadata directory not provided
I20260923 20:09:38.780750 8204 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:09:38.781057 8204 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:09:38.784226 8204 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194121577027-1706-0/minicluster-data/ts-2/data/instance:
uuid: "0ecb00af331c4c6eb682ec0f64fa0e47"
format_stamp: "Formatted at 2026-09-23 20:09:38 on dist-test-slave-th4g"
server_key: "a5da5edc9efa612cdad9dae6607d4e32"
server_key_iv: "e01d02b3ac3d282fe6d389699ecd0263"
server_key_version: "encryptionkey@0"
I20260923 20:09:38.784967 8204 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194121577027-1706-0/minicluster-data/ts-2/wal/instance:
uuid: "0ecb00af331c4c6eb682ec0f64fa0e47"
format_stamp: "Formatted at 2026-09-23 20:09:38 on dist-test-slave-th4g"
server_key: "a5da5edc9efa612cdad9dae6607d4e32"
server_key_iv: "e01d02b3ac3d282fe6d389699ecd0263"
server_key_version: "encryptionkey@0"
I20260923 20:09:38.790510 8204 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260923 20:09:38.794261 8219 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:09:38.795738 8204 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260923 20:09:38.795953 8204 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194121577027-1706-0/minicluster-data/ts-2/data,/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194121577027-1706-0/minicluster-data/ts-2/wal
uuid: "0ecb00af331c4c6eb682ec0f64fa0e47"
format_stamp: "Formatted at 2026-09-23 20:09:38 on dist-test-slave-th4g"
server_key: "a5da5edc9efa612cdad9dae6607d4e32"
server_key_iv: "e01d02b3ac3d282fe6d389699ecd0263"
server_key_version: "encryptionkey@0"
I20260923 20:09:38.796165 8204 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194121577027-1706-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194121577027-1706-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194121577027-1706-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:09:38.829340 8204 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:09:38.833344 8204 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:09:38.833633 8204 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:09:38.834728 8204 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 20:09:38.836422 8204 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 20:09:38.836503 8204 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 20:09:38.836617 8204 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 20:09:38.836679 8204 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 20:09:38.870168 8064 heartbeater.cc:503] Master 127.1.170.190:42111 was elected leader, sending a full tablet report...
I20260923 20:09:38.871022 8204 rpc_server.cc:307] RPC server started. Bound to: 127.1.170.131:34805
I20260923 20:09:38.871090 8332 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.170.131:34805 every 8 connection(s)
I20260923 20:09:38.872345 8204 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194121577027-1706-0/minicluster-data/ts-2/data/info.pb
I20260923 20:09:38.878007 1706 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqUuku6/build/asan/bin/kudu as pid 8204
I20260923 20:09:38.878237 1706 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194121577027-1706-0/minicluster-data/ts-2/wal/instance
I20260923 20:09:38.878645 1706 external_mini_cluster.cc:1428] Setting key 8ff074f6b4d04b06f0f3f0cc4a576418
Sep 23 20:09:38 dist-test-slave-th4g krb5kdc[7830](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194178, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.170.131@KRBTEST.COM for oryx/127.1.170.190@KRBTEST.COM
I20260923 20:09:38.891788 8066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:09:38.874703 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.170.131:48545 (local address 127.1.170.190:42111)
0923 20:09:38.874876 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0923 20:09:38.874881 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 20:09:38.875912 (+ 1031us) server_negotiation.cc:407] Connection header received
0923 20:09:38.877642 (+ 1730us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:09:38.877648 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:09:38.877735 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:09:38.877859 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:09:38.879586 (+ 1727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:09:38.880598 (+ 1012us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:09:38.882338 (+ 1740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:09:38.882731 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:09:38.882914 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:09:38.886170 (+ 3256us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:09:38.886182 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:09:38.886186 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:09:38.886215 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:09:38.888702 (+ 2487us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:09:38.889493 (+ 791us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:09:38.889498 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:09:38.889507 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:09:38.889626 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:09:38.890135 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:09:38.890145 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:09:38.890147 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:09:38.890407 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:09:38.890556 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0923 20:09:38.891234 (+ 678us) server_negotiation.cc:299] Negotiation successful
0923 20:09:38.891438 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260923 20:09:38.892939 8335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:09:38.875122 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.190:42111 (local address 127.1.170.131:48545)
0923 20:09:38.875756 (+ 634us) negotiation.cc:107] Waiting for socket to connect
0923 20:09:38.875800 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0923 20:09:38.877347 (+ 1547us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:09:38.878122 (+ 775us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:09:38.878134 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:09:38.878735 (+ 601us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:09:38.879417 (+ 682us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:09:38.879436 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:09:38.880739 (+ 1303us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:09:38.880745 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:09:38.882196 (+ 1451us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:09:38.882208 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:09:38.883039 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:09:38.883047 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:09:38.883270 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:09:38.883847 (+ 577us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:09:38.883871 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:09:38.885980 (+ 2109us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:09:38.888887 (+ 2907us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:09:38.888896 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:09:38.888914 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:09:38.889345 (+ 431us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:09:38.889748 (+ 403us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:09:38.889754 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:09:38.889758 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:09:38.890019 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:09:38.890532 (+ 513us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:09:38.890540 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:09:38.890877 (+ 337us) client_negotiation.cc:769] Sending connection context
0923 20:09:38.891131 (+ 254us) client_negotiation.cc:240] Negotiation successful
0923 20:09:38.891640 (+ 509us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":432,"thread_start_us":258,"threads_started":1}
I20260923 20:09:38.894340 8333 heartbeater.cc:347] Connected to a master server at 127.1.170.190:42111
I20260923 20:09:38.894631 8333 heartbeater.cc:464] Registering TS with master...
I20260923 20:09:38.895372 8333 heartbeater.cc:511] Master 127.1.170.190:42111 requested a full tablet report, sending...
I20260923 20:09:38.896611 7875 ts_manager.cc:194] Registered new tserver with Master: 0ecb00af331c4c6eb682ec0f64fa0e47 (127.1.170.131:34805)
I20260923 20:09:38.897385 7875 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.1.170.131@KRBTEST.COM'} at 127.1.170.131:48545
I20260923 20:09:38.906397 1706 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.1.170.189@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.1.170.189@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.1.170.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194121577027-1706-0/krb5kdc/oryx_127.1.170.189.keytab.
Entry for principal oryx/127.1.170.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194121577027-1706-0/krb5kdc/oryx_127.1.170.189.keytab.
I20260923 20:09:38.954394 1706 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqUuku6/build/asan/bin/kudu
/tmp/dist-test-taskqUuku6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194121577027-1706-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194121577027-1706-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194121577027-1706-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194121577027-1706-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.170.189:40091
--webserver_interface=127.1.170.189
--webserver_port=0
--keytab_file=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194121577027-1706-0/krb5kdc/oryx_127.1.170.189.keytab
--principal=oryx/127.1.170.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.170.148:44293
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.170.190:42111,127.1.170.189:40091
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194121577027-1706-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194121577027-1706-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194121577027-1706-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:09:39.274976 8338 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:09:39.275337 8338 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:09:39.275493 8338 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:09:39.283058 8338 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 20:09:39.283212 8338 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:09:39.283311 8338 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:09:39.283413 8338 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 20:09:39.283485 8338 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 20:09:39.294587 8338 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.170.148:44293
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194121577027-1706-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194121577027-1706-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.1.170.190:42111,127.1.170.189:40091
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194121577027-1706-0/krb5kdc/oryx_127.1.170.189.keytab
--principal=oryx/127.1.170.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.170.189:40091
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194121577027-1706-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.1.170.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194121577027-1706-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
W20260923 20:09:39.294987 8338 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.1.170.190:42111,127.1.170.189:40091'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260923 20:09:39.296069 8338 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:09:39.297384 8338 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:09:39.309559 8343 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:09:39.309664 8344 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:09:39.309598 8346 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:09:39.310590 8338 server_base.cc:1034] running on GCE node
I20260923 20:09:39.311383 8338 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:09:39.312829 8338 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:09:39.314065 8338 hybrid_clock.cc:648] HybridClock initialized: now 1790194179314011 us; error 33 us; skew 500 ppm
I20260923 20:09:39.314427 8338 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:09:39 dist-test-slave-th4g krb5kdc[7830](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194179, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.170.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:09:39.317613 8338 init.cc:377] Logged in from keytab as oryx/127.1.170.189@KRBTEST.COM (short username oryx)
I20260923 20:09:39.319175 8338 webserver.cc:460] Webserver started at http://127.1.170.189:38881/ using document root <none> and password file <none>
I20260923 20:09:39.319885 8338 fs_manager.cc:359] Metadata directory not provided
I20260923 20:09:39.320019 8338 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:09:39.320334 8338 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:09:39.323370 8338 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194121577027-1706-0/minicluster-data/master-1/data/instance:
uuid: "2fc4ff3aee844aa492e3b79e53ad59c3"
format_stamp: "Formatted at 2026-09-23 20:09:39 on dist-test-slave-th4g"
server_key: "3f6403a1550b86e5ee19089d344857ee"
server_key_iv: "fa7125ac2e97169a0e300d625c6d9940"
server_key_version: "encryptionkey@0"
I20260923 20:09:39.324141 8338 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194121577027-1706-0/minicluster-data/master-1/wal/instance:
uuid: "2fc4ff3aee844aa492e3b79e53ad59c3"
format_stamp: "Formatted at 2026-09-23 20:09:39 on dist-test-slave-th4g"
server_key: "3f6403a1550b86e5ee19089d344857ee"
server_key_iv: "fa7125ac2e97169a0e300d625c6d9940"
server_key_version: "encryptionkey@0"
I20260923 20:09:39.329288 8338 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260923 20:09:39.332815 8353 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:09:39.334481 8338 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 20:09:39.334719 8338 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194121577027-1706-0/minicluster-data/master-1/data,/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194121577027-1706-0/minicluster-data/master-1/wal
uuid: "2fc4ff3aee844aa492e3b79e53ad59c3"
format_stamp: "Formatted at 2026-09-23 20:09:39 on dist-test-slave-th4g"
server_key: "3f6403a1550b86e5ee19089d344857ee"
server_key_iv: "fa7125ac2e97169a0e300d625c6d9940"
server_key_version: "encryptionkey@0"
I20260923 20:09:39.334982 8338 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194121577027-1706-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194121577027-1706-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194121577027-1706-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:09:39.358467 8338 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:09:39.363237 8338 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:09:39.363512 8338 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:09:39.383144 8338 rpc_server.cc:307] RPC server started. Bound to: 127.1.170.189:40091
I20260923 20:09:39.383126 8405 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.170.189:40091 every 8 connection(s)
I20260923 20:09:39.384670 8338 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194121577027-1706-0/minicluster-data/master-1/data/info.pb
I20260923 20:09:39.386503 8199 heartbeater.cc:503] Master 127.1.170.190:42111 was elected leader, sending a full tablet report...
I20260923 20:09:39.388747 8406 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:09:39.395056 1706 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqUuku6/build/asan/bin/kudu as pid 8338
I20260923 20:09:39.395424 8406 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.170.190" port: 42111 } has no permanent_uuid. Determining permanent_uuid...
Sep 23 20:09:39 dist-test-slave-th4g krb5kdc[7830](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194179, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.170.189@KRBTEST.COM for oryx/127.1.170.190@KRBTEST.COM
Sep 23 20:09:39 dist-test-slave-th4g krb5kdc[7830](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194176, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.1.170.189@KRBTEST.COM
I20260923 20:09:39.416414 8410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:09:39.398043 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41838 (local address 127.1.170.190:42111)
0923 20:09:39.398553 (+ 510us) server_negotiation.cc:206] Beginning negotiation
0923 20:09:39.398560 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:09:39.399082 (+ 522us) server_negotiation.cc:407] Connection header received
0923 20:09:39.400520 (+ 1438us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:09:39.400524 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:09:39.400626 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:09:39.400782 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:09:39.402184 (+ 1402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:09:39.403034 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:09:39.404981 (+ 1947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:09:39.405540 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:09:39.405637 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:09:39.409501 (+ 3864us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:09:39.409509 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:09:39.409515 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:09:39.409563 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:09:39.413086 (+ 3523us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:09:39.414012 (+ 926us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:09:39.414018 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:09:39.414022 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:09:39.414195 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:09:39.414623 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:09:39.414632 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:09:39.414635 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:09:39.414956 (+ 321us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:09:39.415113 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0923 20:09:39.415819 (+ 706us) server_negotiation.cc:299] Negotiation successful
0923 20:09:39.416029 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":377,"thread_start_us":238,"threads_started":1}
I20260923 20:09:39.417589 8409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:09:39.397145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45324 (local address 127.1.170.189:40091)
0923 20:09:39.397962 (+ 817us) server_negotiation.cc:206] Beginning negotiation
0923 20:09:39.397978 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0923 20:09:39.398027 (+ 49us) server_negotiation.cc:407] Connection header received
0923 20:09:39.400214 (+ 2187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:09:39.400242 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:09:39.400541 (+ 299us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:09:39.400700 (+ 159us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:09:39.402510 (+ 1810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:09:39.403991 (+ 1481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:09:39.405348 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:09:39.406034 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:09:39.406163 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:09:39.410560 (+ 4397us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:09:39.410570 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:09:39.410587 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:09:39.410632 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:09:39.414292 (+ 3660us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:09:39.414968 (+ 676us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:09:39.414976 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:09:39.414980 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:09:39.415105 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:09:39.415475 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:09:39.415482 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:09:39.415485 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:09:39.415829 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:09:39.416130 (+ 301us) server_negotiation.cc:1035] Waiting for connection context
0923 20:09:39.416445 (+ 315us) server_negotiation.cc:299] Negotiation successful
0923 20:09:39.416847 (+ 402us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":562,"thread_start_us":369,"threads_started":1}
I20260923 20:09:39.417637 8411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:09:39.398051 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.190:42111 (local address 127.0.0.1:41838)
0923 20:09:39.398918 (+ 867us) negotiation.cc:107] Waiting for socket to connect
0923 20:09:39.398967 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0923 20:09:39.400214 (+ 1247us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:09:39.400864 (+ 650us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:09:39.400870 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:09:39.401594 (+ 724us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:09:39.401997 (+ 403us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:09:39.402015 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:09:39.403181 (+ 1166us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:09:39.403187 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:09:39.404779 (+ 1592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:09:39.404790 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:09:39.405758 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:09:39.405765 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:09:39.406030 (+ 265us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:09:39.406863 (+ 833us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:09:39.406895 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:09:39.409290 (+ 2395us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:09:39.413255 (+ 3965us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:09:39.413264 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:09:39.413274 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:09:39.413815 (+ 541us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:09:39.414313 (+ 498us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:09:39.414317 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:09:39.414319 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:09:39.414490 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:09:39.415098 (+ 608us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:09:39.415106 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:09:39.415437 (+ 331us) client_negotiation.cc:769] Sending connection context
0923 20:09:39.415726 (+ 289us) client_negotiation.cc:240] Negotiation successful
0923 20:09:39.416219 (+ 493us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":607,"thread_start_us":458,"threads_started":1}
I20260923 20:09:39.420393 8406 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.170.189" port: 40091 } has no permanent_uuid. Determining permanent_uuid...
Sep 23 20:09:39 dist-test-slave-th4g krb5kdc[7830](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194179, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.170.189@KRBTEST.COM for oryx/127.1.170.189@KRBTEST.COM
I20260923 20:09:39.432081 8411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:09:39.421065 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.189:40091 (local address 127.0.0.1:45334)
0923 20:09:39.421288 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0923 20:09:39.421306 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0923 20:09:39.421401 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:09:39.421687 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:09:39.421690 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:09:39.422065 (+ 375us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:09:39.422196 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:09:39.422203 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:09:39.423155 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:09:39.423158 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:09:39.424019 (+ 861us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:09:39.424029 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:09:39.424672 (+ 643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:09:39.424677 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:09:39.424824 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:09:39.425435 (+ 611us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:09:39.425451 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:09:39.427480 (+ 2029us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:09:39.430188 (+ 2708us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:09:39.430191 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:09:39.430215 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:09:39.430559 (+ 344us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:09:39.430867 (+ 308us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:09:39.430870 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:09:39.430872 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:09:39.430986 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:09:39.431430 (+ 444us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:09:39.431433 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:09:39.431547 (+ 114us) client_negotiation.cc:769] Sending connection context
0923 20:09:39.431650 (+ 103us) client_negotiation.cc:240] Negotiation successful
0923 20:09:39.431848 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":56}
I20260923 20:09:39.432264 8409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:09:39.421328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45334 (local address 127.1.170.189:40091)
0923 20:09:39.421468 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0923 20:09:39.421472 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 20:09:39.421486 (+ 14us) server_negotiation.cc:407] Connection header received
0923 20:09:39.421522 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:09:39.421525 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:09:39.421574 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:09:39.421651 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:09:39.422343 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:09:39.423034 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:09:39.424134 (+ 1100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:09:39.424528 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:09:39.424630 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:09:39.427655 (+ 3025us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:09:39.427660 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:09:39.427663 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:09:39.427689 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:09:39.430060 (+ 2371us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:09:39.430656 (+ 596us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:09:39.430660 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:09:39.430662 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:09:39.430775 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:09:39.431070 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:09:39.431073 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:09:39.431075 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:09:39.431336 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:09:39.431439 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0923 20:09:39.431744 (+ 305us) server_negotiation.cc:299] Negotiation successful
0923 20:09:39.431941 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":49,"server-negotiator.queue_time_us":51}
I20260923 20:09:39.436874 8406 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2fc4ff3aee844aa492e3b79e53ad59c3: Bootstrap starting.
I20260923 20:09:39.440152 8406 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2fc4ff3aee844aa492e3b79e53ad59c3: Neither blocks nor log segments found. Creating new log.
I20260923 20:09:39.441216 8406 log.cc:824] T 00000000000000000000000000000000 P 2fc4ff3aee844aa492e3b79e53ad59c3: Log is configured to *not* fsync() on all Append() calls
I20260923 20:09:39.443915 8406 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2fc4ff3aee844aa492e3b79e53ad59c3: No bootstrap required, opened a new log
I20260923 20:09:39.450799 8406 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2fc4ff3aee844aa492e3b79e53ad59c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f83afa4c3fa644f08196bf2ef04d733b" member_type: VOTER last_known_addr { host: "127.1.170.190" port: 42111 } } peers { permanent_uuid: "2fc4ff3aee844aa492e3b79e53ad59c3" member_type: VOTER last_known_addr { host: "127.1.170.189" port: 40091 } }
I20260923 20:09:39.451169 8406 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2fc4ff3aee844aa492e3b79e53ad59c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:09:39.451280 8406 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2fc4ff3aee844aa492e3b79e53ad59c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2fc4ff3aee844aa492e3b79e53ad59c3, State: Initialized, Role: FOLLOWER
I20260923 20:09:39.451861 8406 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2fc4ff3aee844aa492e3b79e53ad59c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f83afa4c3fa644f08196bf2ef04d733b" member_type: VOTER last_known_addr { host: "127.1.170.190" port: 42111 } } peers { permanent_uuid: "2fc4ff3aee844aa492e3b79e53ad59c3" member_type: VOTER last_known_addr { host: "127.1.170.189" port: 40091 } }
I20260923 20:09:39.452859 8413 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2fc4ff3aee844aa492e3b79e53ad59c3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f83afa4c3fa644f08196bf2ef04d733b" member_type: VOTER last_known_addr { host: "127.1.170.190" port: 42111 } } peers { permanent_uuid: "2fc4ff3aee844aa492e3b79e53ad59c3" member_type: VOTER last_known_addr { host: "127.1.170.189" port: 40091 } } }
I20260923 20:09:39.453178 8413 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2fc4ff3aee844aa492e3b79e53ad59c3 [sys.catalog]: This master's current role is: FOLLOWER
I20260923 20:09:39.453783 8406 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2fc4ff3aee844aa492e3b79e53ad59c3 [sys.catalog]: configured and running, proceeding with master startup.
W20260923 20:09:39.462255 8424 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 2fc4ff3aee844aa492e3b79e53ad59c3: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260923 20:09:39.462412 8424 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260923 20:09:39.465308 8338 master_runner.cc:428] Detected that this master 2fc4ff3aee844aa492e3b79e53ad59c3 is joining an existing cluster
I20260923 20:09:39.465406 8338 master_runner.cc:432] Initiating AddMaster RPC to add 127.1.170.189:40091
I20260923 20:09:39.486419 8409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:09:39.469399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45338 (local address 127.1.170.189:40091)
0923 20:09:39.469605 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0923 20:09:39.469610 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 20:09:39.471174 (+ 1564us) server_negotiation.cc:407] Connection header received
0923 20:09:39.471406 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:09:39.471412 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:09:39.471474 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:09:39.471539 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:09:39.472915 (+ 1376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:09:39.474117 (+ 1202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:09:39.476202 (+ 2085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:09:39.477027 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:09:39.477166 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:09:39.479953 (+ 2787us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:09:39.479962 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:09:39.479968 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:09:39.480006 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:09:39.483435 (+ 3429us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:09:39.484200 (+ 765us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:09:39.484205 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:09:39.484209 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:09:39.484374 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:09:39.484774 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:09:39.484779 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:09:39.484782 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:09:39.485139 (+ 357us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:09:39.485288 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0923 20:09:39.485765 (+ 477us) server_negotiation.cc:299] Negotiation successful
0923 20:09:39.486014 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":22,"server-negotiator.queue_time_us":35}
I20260923 20:09:39.487103 8410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:09:39.469165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41848 (local address 127.1.170.190:42111)
0923 20:09:39.469381 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0923 20:09:39.469387 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:09:39.470362 (+ 975us) server_negotiation.cc:407] Connection header received
0923 20:09:39.470551 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:09:39.470556 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:09:39.470652 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:09:39.470772 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:09:39.472334 (+ 1562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:09:39.473458 (+ 1124us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:09:39.475576 (+ 2118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:09:39.476215 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:09:39.476315 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:09:39.479968 (+ 3653us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:09:39.479975 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:09:39.479980 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:09:39.480019 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:09:39.483878 (+ 3859us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:09:39.484688 (+ 810us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:09:39.484693 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:09:39.484697 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:09:39.484839 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:09:39.485284 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:09:39.485291 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:09:39.485296 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:09:39.485648 (+ 352us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:09:39.485797 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0923 20:09:39.486469 (+ 672us) server_negotiation.cc:299] Negotiation successful
0923 20:09:39.486699 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260923 20:09:39.487445 8431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:09:39.470399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.189:40091 (local address 127.0.0.1:45338)
0923 20:09:39.471088 (+ 689us) negotiation.cc:107] Waiting for socket to connect
0923 20:09:39.471109 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0923 20:09:39.471265 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:09:39.471712 (+ 447us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:09:39.471716 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:09:39.472445 (+ 729us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:09:39.472755 (+ 310us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:09:39.472768 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:09:39.474356 (+ 1588us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:09:39.474364 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:09:39.476060 (+ 1696us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:09:39.476076 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:09:39.477237 (+ 1161us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:09:39.477249 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:09:39.477544 (+ 295us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:09:39.478357 (+ 813us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:09:39.478388 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:09:39.479779 (+ 1391us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:09:39.483559 (+ 3780us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:09:39.483567 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:09:39.483574 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:09:39.484065 (+ 491us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:09:39.484503 (+ 438us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:09:39.484508 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:09:39.484512 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:09:39.484668 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:09:39.485372 (+ 704us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:09:39.485376 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:09:39.485523 (+ 147us) client_negotiation.cc:769] Sending connection context
0923 20:09:39.486676 (+ 1153us) client_negotiation.cc:240] Negotiation successful
0923 20:09:39.486958 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":555,"mutex_wait_us":40,"thread_start_us":7490,"threads_started":1}
I20260923 20:09:39.488306 8430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:09:39.469411 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.190:42111 (local address 127.0.0.1:41848)
0923 20:09:39.470206 (+ 795us) negotiation.cc:107] Waiting for socket to connect
0923 20:09:39.470235 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0923 20:09:39.470414 (+ 179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:09:39.470972 (+ 558us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:09:39.470977 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:09:39.471625 (+ 648us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:09:39.472161 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:09:39.472178 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:09:39.473643 (+ 1465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:09:39.473648 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:09:39.475401 (+ 1753us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:09:39.475414 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:09:39.476600 (+ 1186us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:09:39.476607 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:09:39.477464 (+ 857us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:09:39.478445 (+ 981us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:09:39.478467 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:09:39.479749 (+ 1282us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:09:39.484028 (+ 4279us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:09:39.484036 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:09:39.484043 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:09:39.484547 (+ 504us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:09:39.484977 (+ 430us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:09:39.484984 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:09:39.484990 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:09:39.485153 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:09:39.485895 (+ 742us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:09:39.485902 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:09:39.486102 (+ 200us) client_negotiation.cc:769] Sending connection context
0923 20:09:39.487349 (+ 1247us) client_negotiation.cc:240] Negotiation successful
0923 20:09:39.487722 (+ 373us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":634,"mutex_wait_us":78,"thread_start_us":458,"threads_started":1}
WARNING: no policy specified for oryx/127.1.170.188@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.1.170.188@KRBTEST.COM" created.
Sep 23 20:09:39 dist-test-slave-th4g krb5kdc[7830](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194177, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.170.190@KRBTEST.COM for oryx/127.1.170.189@KRBTEST.COM
I20260923 20:09:39.513238 8433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:09:39.495175 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.189:40091 (local address 127.0.0.1:45344)
0923 20:09:39.495965 (+ 790us) negotiation.cc:107] Waiting for socket to connect
0923 20:09:39.496014 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0923 20:09:39.496235 (+ 221us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:09:39.496806 (+ 571us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:09:39.496816 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:09:39.497683 (+ 867us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:09:39.498005 (+ 322us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:09:39.498015 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:09:39.499438 (+ 1423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:09:39.499442 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:09:39.500904 (+ 1462us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:09:39.500922 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:09:39.501995 (+ 1073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:09:39.502003 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:09:39.502254 (+ 251us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:09:39.503027 (+ 773us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:09:39.503055 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:09:39.506041 (+ 2986us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:09:39.510016 (+ 3975us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:09:39.510031 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:09:39.510039 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:09:39.510631 (+ 592us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:09:39.511077 (+ 446us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:09:39.511083 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:09:39.511087 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:09:39.511251 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:09:39.511943 (+ 692us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:09:39.511954 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:09:39.512170 (+ 216us) client_negotiation.cc:769] Sending connection context
0923 20:09:39.512433 (+ 263us) client_negotiation.cc:240] Negotiation successful
0923 20:09:39.512790 (+ 357us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":530,"thread_start_us":364,"threads_started":1}
I20260923 20:09:39.513278 8409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:09:39.495235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45344 (local address 127.1.170.189:40091)
0923 20:09:39.495433 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0923 20:09:39.495440 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:09:39.496092 (+ 652us) server_negotiation.cc:407] Connection header received
0923 20:09:39.496484 (+ 392us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:09:39.496492 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:09:39.496624 (+ 132us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:09:39.496730 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:09:39.498164 (+ 1434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:09:39.499308 (+ 1144us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:09:39.501059 (+ 1751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:09:39.501770 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:09:39.501940 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:09:39.506250 (+ 4310us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:09:39.506259 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:09:39.506265 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:09:39.506310 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:09:39.509863 (+ 3553us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:09:39.510766 (+ 903us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:09:39.510774 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:09:39.510781 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:09:39.510944 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:09:39.511390 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:09:39.511398 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:09:39.511403 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:09:39.511794 (+ 391us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:09:39.511947 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0923 20:09:39.512546 (+ 599us) server_negotiation.cc:299] Negotiation successful
0923 20:09:39.512804 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260923 20:09:39.515823 7875 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.1.170.189:40091: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "2fc4ff3aee844aa492e3b79e53ad59c3"
member_type: NON_VOTER
last_known_addr {
host: "127.1.170.189"
port: 40091
}
attrs {
promote: true
}
}
dest_uuid: "f83afa4c3fa644f08196bf2ef04d733b"
cas_config_opid_index: -1
I20260923 20:09:39.517375 7875 consensus_queue.cc:237] T 00000000000000000000000000000000 P f83afa4c3fa644f08196bf2ef04d733b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "f83afa4c3fa644f08196bf2ef04d733b" member_type: VOTER last_known_addr { host: "127.1.170.190" port: 42111 } } peers { permanent_uuid: "2fc4ff3aee844aa492e3b79e53ad59c3" member_type: NON_VOTER last_known_addr { host: "127.1.170.189" port: 40091 } attrs { promote: true } }
I20260923 20:09:39.522069 8437 raft_consensus.cc:2932] T 00000000000000000000000000000000 P f83afa4c3fa644f08196bf2ef04d733b [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 2fc4ff3aee844aa492e3b79e53ad59c3 (127.1.170.189) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "f83afa4c3fa644f08196bf2ef04d733b" member_type: VOTER last_known_addr { host: "127.1.170.190" port: 42111 } } peers { permanent_uuid: "2fc4ff3aee844aa492e3b79e53ad59c3" member_type: NON_VOTER last_known_addr { host: "127.1.170.189" port: 40091 } attrs { promote: true } } }
I20260923 20:09:39.523052 8380 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2fc4ff3aee844aa492e3b79e53ad59c3 [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:09:39.526659 8437 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.1.170.189:40091
I20260923 20:09:39.527716 8338 master.cc:537] Master@127.1.170.189:40091 shutting down...
I20260923 20:09:39.528844 8380 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 2fc4ff3aee844aa492e3b79e53ad59c3 [term 1 FOLLOWER]: Refusing update from remote peer f83afa4c3fa644f08196bf2ef04d733b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.1.170.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194121577027-1706-0/krb5kdc/oryx_127.1.170.188.keytab.
Entry for principal oryx/127.1.170.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194121577027-1706-0/krb5kdc/oryx_127.1.170.188.keytab.
I20260923 20:09:39.531131 1706 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqUuku6/build/asan/bin/kudu
/tmp/dist-test-taskqUuku6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194121577027-1706-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194121577027-1706-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194121577027-1706-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194121577027-1706-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.170.188:36059
--webserver_interface=127.1.170.188
--webserver_port=0
--keytab_file=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194121577027-1706-0/krb5kdc/oryx_127.1.170.188.keytab
--principal=oryx/127.1.170.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.170.148:44293
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.170.190:42111,127.1.170.189:40091,127.1.170.188:36059
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194121577027-1706-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194121577027-1706-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194121577027-1706-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260923 20:09:39.533946 8436 sys_catalog.cc:455] T 00000000000000000000000000000000 P f83afa4c3fa644f08196bf2ef04d733b [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "f83afa4c3fa644f08196bf2ef04d733b" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "f83afa4c3fa644f08196bf2ef04d733b" member_type: VOTER last_known_addr { host: "127.1.170.190" port: 42111 } } peers { permanent_uuid: "2fc4ff3aee844aa492e3b79e53ad59c3" member_type: NON_VOTER last_known_addr { host: "127.1.170.189" port: 40091 } attrs { promote: true } } }
I20260923 20:09:39.534289 8436 sys_catalog.cc:458] T 00000000000000000000000000000000 P f83afa4c3fa644f08196bf2ef04d733b [sys.catalog]: This master's current role is: LEADER
I20260923 20:09:39.537369 8437 consensus_queue.cc:1035] T 00000000000000000000000000000000 P f83afa4c3fa644f08196bf2ef04d733b [LEADER]: Connected to new peer: Peer: permanent_uuid: "2fc4ff3aee844aa492e3b79e53ad59c3" member_type: NON_VOTER last_known_addr { host: "127.1.170.189" port: 40091 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
W20260923 20:09:39.540505 7862 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
I20260923 20:09:39.547365 8338 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 2fc4ff3aee844aa492e3b79e53ad59c3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 20:09:39.547803 8338 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 2fc4ff3aee844aa492e3b79e53ad59c3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 20:09:39.547974 8338 tablet_replica.cc:330] T 00000000000000000000000000000000 P 2fc4ff3aee844aa492e3b79e53ad59c3: stopping tablet replica
W20260923 20:09:39.550566 7862 consensus_peers.cc:487] T 00000000000000000000000000000000 P f83afa4c3fa644f08196bf2ef04d733b -> Peer 2fc4ff3aee844aa492e3b79e53ad59c3 (127.1.170.189:40091): Couldn't send request to peer 2fc4ff3aee844aa492e3b79e53ad59c3. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260923 20:09:39.555099 8338 master.cc:552] Master@127.1.170.189:40091 shutdown complete.
I20260923 20:09:39.555184 8338 master_runner.cc:305] Clearing existing system tablet
I20260923 20:09:39.555670 8338 fs_manager.cc:359] Metadata directory not provided
I20260923 20:09:39.555773 8338 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:09:39.558709 8338 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.004s sys 0.000s
I20260923 20:09:39.560477 8442 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:09:39.560837 8338 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194121577027-1706-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:09:39.561339 8338 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260923 20:09:39.561477 8338 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194121577027-1706-0/minicluster-data/master-1/data,/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194121577027-1706-0/minicluster-data/master-1/wal
uuid: "2fc4ff3aee844aa492e3b79e53ad59c3"
format_stamp: "Formatted at 2026-09-23 20:09:39 on dist-test-slave-th4g"
server_key: "3f6403a1550b86e5ee19089d344857ee"
server_key_iv: "fa7125ac2e97169a0e300d625c6d9940"
server_key_version: "encryptionkey@0"
I20260923 20:09:39.567051 8338 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 2fc4ff3aee844aa492e3b79e53ad59c3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 20:09:39.574036 8338 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 2fc4ff3aee844aa492e3b79e53ad59c3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260923 20:09:39.574324 8338 log.cc:1196] T 00000000000000000000000000000000 P 2fc4ff3aee844aa492e3b79e53ad59c3: Deleting WAL directory at /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194121577027-1706-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260923 20:09:39.574828 8338 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 2fc4ff3aee844aa492e3b79e53ad59c3: Deleting consensus metadata
I20260923 20:09:39.575135 8338 master_runner.cc:315] Copying system tablet from 127.1.170.190:42111
I20260923 20:09:39.577934 8338 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 2fc4ff3aee844aa492e3b79e53ad59c3: tablet copy: Beginning tablet copy session from remote peer at address 127.1.170.190:42111
I20260923 20:09:39.597558 8410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:09:39.578913 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41856 (local address 127.1.170.190:42111)
0923 20:09:39.579094 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0923 20:09:39.579101 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:09:39.582461 (+ 3360us) server_negotiation.cc:407] Connection header received
0923 20:09:39.582732 (+ 271us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:09:39.582738 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:09:39.582831 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:09:39.582950 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:09:39.583941 (+ 991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:09:39.585084 (+ 1143us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:09:39.587000 (+ 1916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:09:39.587668 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:09:39.587802 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:09:39.590313 (+ 2511us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:09:39.590322 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:09:39.590330 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:09:39.590386 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:09:39.594585 (+ 4199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:09:39.595418 (+ 833us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:09:39.595424 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:09:39.595430 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:09:39.595582 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:09:39.596010 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:09:39.596015 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:09:39.596017 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:09:39.596386 (+ 369us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:09:39.596539 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0923 20:09:39.596995 (+ 456us) server_negotiation.cc:299] Negotiation successful
0923 20:09:39.597219 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260923 20:09:39.598084 8448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:09:39.578807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.190:42111 (local address 127.0.0.1:41856)
0923 20:09:39.582309 (+ 3502us) negotiation.cc:107] Waiting for socket to connect
0923 20:09:39.582331 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0923 20:09:39.582499 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:09:39.582995 (+ 496us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:09:39.583000 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:09:39.583505 (+ 505us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:09:39.583773 (+ 268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:09:39.583786 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:09:39.585265 (+ 1479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:09:39.585277 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:09:39.586823 (+ 1546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:09:39.586845 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:09:39.587833 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:09:39.587840 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:09:39.588023 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:09:39.588858 (+ 835us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:09:39.588882 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:09:39.590023 (+ 1141us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:09:39.594741 (+ 4718us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:09:39.594747 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:09:39.594752 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:09:39.595259 (+ 507us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:09:39.595699 (+ 440us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:09:39.595704 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:09:39.595708 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:09:39.595880 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:09:39.596518 (+ 638us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:09:39.596524 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:09:39.596708 (+ 184us) client_negotiation.cc:769] Sending connection context
0923 20:09:39.596854 (+ 146us) client_negotiation.cc:240] Negotiation successful
0923 20:09:39.597172 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3295,"thread_start_us":3202,"threads_started":1}
I20260923 20:09:39.598989 7895 tablet_copy_service.cc:140] P f83afa4c3fa644f08196bf2ef04d733b: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 2fc4ff3aee844aa492e3b79e53ad59c3 ({username='oryx', principal='oryx/127.1.170.189@KRBTEST.COM'} at 127.0.0.1:41856)
I20260923 20:09:39.599478 7895 tablet_copy_service.cc:161] P f83afa4c3fa644f08196bf2ef04d733b: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 2fc4ff3aee844aa492e3b79e53ad59c3 at {username='oryx', principal='oryx/127.1.170.189@KRBTEST.COM'} at 127.0.0.1:41856: session id = 2fc4ff3aee844aa492e3b79e53ad59c3-00000000000000000000000000000000
I20260923 20:09:39.604523 7895 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P f83afa4c3fa644f08196bf2ef04d733b: Tablet Copy: opened 0 blocks and 1 log segments
I20260923 20:09:39.606894 8338 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260923 20:09:39.614303 8338 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 2fc4ff3aee844aa492e3b79e53ad59c3: tablet copy: Starting download of 0 data blocks...
I20260923 20:09:39.614714 8338 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 2fc4ff3aee844aa492e3b79e53ad59c3: tablet copy: Starting download of 1 WAL segments...
I20260923 20:09:39.618436 8338 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 2fc4ff3aee844aa492e3b79e53ad59c3: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260923 20:09:39.623230 7895 tablet_copy_service.cc:342] P f83afa4c3fa644f08196bf2ef04d733b: Request end of tablet copy session 2fc4ff3aee844aa492e3b79e53ad59c3-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.1.170.189@KRBTEST.COM'} at 127.0.0.1:41856
I20260923 20:09:39.623518 7895 tablet_copy_service.cc:433] P f83afa4c3fa644f08196bf2ef04d733b: ending tablet copy session 2fc4ff3aee844aa492e3b79e53ad59c3-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 2fc4ff3aee844aa492e3b79e53ad59c3
I20260923 20:09:39.634231 8338 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:09:39.636914 8449 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:09:39.642657 8450 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:09:39.655954 8338 server_base.cc:1034] running on GCE node
W20260923 20:09:39.655946 8452 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:09:39.656571 8338 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:09:39.657305 8338 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:09:39.658519 8338 hybrid_clock.cc:648] HybridClock initialized: now 1790194179658450 us; error 40 us; skew 500 ppm
I20260923 20:09:39.658763 8338 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:09:39 dist-test-slave-th4g krb5kdc[7830](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194179, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.170.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:09:39.660933 8338 init.cc:377] Logged in from keytab as oryx/127.1.170.189@KRBTEST.COM (short username oryx)
I20260923 20:09:39.665566 8338 webserver.cc:460] Webserver started at http://127.1.170.189:44831/ using document root <none> and password file <none>
I20260923 20:09:39.666040 8338 fs_manager.cc:359] Metadata directory not provided
I20260923 20:09:39.666126 8338 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:09:39.669288 8338 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260923 20:09:39.671238 8459 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:09:39.671700 8338 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260923 20:09:39.671881 8338 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194121577027-1706-0/minicluster-data/master-1/data,/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194121577027-1706-0/minicluster-data/master-1/wal
uuid: "2fc4ff3aee844aa492e3b79e53ad59c3"
format_stamp: "Formatted at 2026-09-23 20:09:39 on dist-test-slave-th4g"
server_key: "3f6403a1550b86e5ee19089d344857ee"
server_key_iv: "fa7125ac2e97169a0e300d625c6d9940"
server_key_version: "encryptionkey@0"
I20260923 20:09:39.672115 8338 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194121577027-1706-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194121577027-1706-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194121577027-1706-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:09:39.681779 8338 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:09:39.686204 8338 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:09:39.716697 8338 rpc_server.cc:307] RPC server started. Bound to: 127.1.170.189:40091
I20260923 20:09:39.716676 8510 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.170.189:40091 every 8 connection(s)
I20260923 20:09:39.717639 8338 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194121577027-1706-0/minicluster-data/master-1/data/info.pb
I20260923 20:09:39.724696 8511 sys_catalog.cc:263] Verifying existing consensus state
I20260923 20:09:39.727331 8511 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2fc4ff3aee844aa492e3b79e53ad59c3: Bootstrap starting.
I20260923 20:09:39.741148 8511 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2fc4ff3aee844aa492e3b79e53ad59c3: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260923 20:09:39.741735 8511 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2fc4ff3aee844aa492e3b79e53ad59c3: Bootstrap complete.
I20260923 20:09:39.742942 8511 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2fc4ff3aee844aa492e3b79e53ad59c3 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "f83afa4c3fa644f08196bf2ef04d733b" member_type: VOTER last_known_addr { host: "127.1.170.190" port: 42111 } } peers { permanent_uuid: "2fc4ff3aee844aa492e3b79e53ad59c3" member_type: NON_VOTER last_known_addr { host: "127.1.170.189" port: 40091 } attrs { promote: true } }
I20260923 20:09:39.743278 8511 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2fc4ff3aee844aa492e3b79e53ad59c3 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 2fc4ff3aee844aa492e3b79e53ad59c3, State: Initialized, Role: LEARNER
I20260923 20:09:39.743566 8511 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2fc4ff3aee844aa492e3b79e53ad59c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "f83afa4c3fa644f08196bf2ef04d733b" member_type: VOTER last_known_addr { host: "127.1.170.190" port: 42111 } } peers { permanent_uuid: "2fc4ff3aee844aa492e3b79e53ad59c3" member_type: NON_VOTER last_known_addr { host: "127.1.170.189" port: 40091 } attrs { promote: true } }
I20260923 20:09:39.744405 8514 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2fc4ff3aee844aa492e3b79e53ad59c3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "f83afa4c3fa644f08196bf2ef04d733b" member_type: VOTER last_known_addr { host: "127.1.170.190" port: 42111 } } peers { permanent_uuid: "2fc4ff3aee844aa492e3b79e53ad59c3" member_type: NON_VOTER last_known_addr { host: "127.1.170.189" port: 40091 } attrs { promote: true } } }
I20260923 20:09:39.744838 8511 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2fc4ff3aee844aa492e3b79e53ad59c3 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 20:09:39.744891 8514 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2fc4ff3aee844aa492e3b79e53ad59c3 [sys.catalog]: This master's current role is: LEARNER
I20260923 20:09:39.753312 8525 catalog_manager.cc:1259] Loaded cluster ID: 62f93adc2448407bb12c561910d92d81
I20260923 20:09:39.753451 8525 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 2fc4ff3aee844aa492e3b79e53ad59c3: loading cluster ID for follower catalog manager: success
I20260923 20:09:39.757299 8525 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 2fc4ff3aee844aa492e3b79e53ad59c3: acquiring CA information for follower catalog manager: success
I20260923 20:09:39.760874 8525 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 2fc4ff3aee844aa492e3b79e53ad59c3: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260923 20:09:39.899879 8333 heartbeater.cc:503] Master 127.1.170.190:42111 was elected leader, sending a full tablet report...
I20260923 20:09:39.936014 8433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:09:39.926771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.189:40091 (local address 127.0.0.1:45356)
0923 20:09:39.927040 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0923 20:09:39.927060 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0923 20:09:39.927185 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:09:39.928349 (+ 1164us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:09:39.928356 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:09:39.928394 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:09:39.928633 (+ 239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:09:39.928646 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:09:39.930736 (+ 2090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:09:39.930742 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:09:39.933143 (+ 2401us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:09:39.933166 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:09:39.935089 (+ 1923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:09:39.935096 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:09:39.935330 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:09:39.935390 (+ 60us) client_negotiation.cc:769] Sending connection context
0923 20:09:39.935537 (+ 147us) client_negotiation.cc:240] Negotiation successful
0923 20:09:39.935699 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145}
I20260923 20:09:39.936275 8526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:09:39.927001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45356 (local address 127.1.170.189:40091)
0923 20:09:39.927759 (+ 758us) server_negotiation.cc:206] Beginning negotiation
0923 20:09:39.927765 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:09:39.927785 (+ 20us) server_negotiation.cc:407] Connection header received
0923 20:09:39.927857 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:09:39.927862 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:09:39.927961 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:09:39.928101 (+ 140us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:09:39.928821 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:09:39.930542 (+ 1721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:09:39.933367 (+ 2825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:09:39.934896 (+ 1529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:09:39.935056 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:09:39.935421 (+ 365us) server_negotiation.cc:1035] Waiting for connection context
0923 20:09:39.935584 (+ 163us) server_negotiation.cc:299] Negotiation successful
0923 20:09:39.935735 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":604,"thread_start_us":412,"threads_started":1}
I20260923 20:09:39.937636 8486 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 2fc4ff3aee844aa492e3b79e53ad59c3 [term 1 LEARNER]: Deduplicated request from leader. Original: 0.0->[1.1-1.5] Dedup: 1.5->[]
I20260923 20:09:39.938592 8514 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2fc4ff3aee844aa492e3b79e53ad59c3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f83afa4c3fa644f08196bf2ef04d733b. Latest consensus state: current_term: 1 leader_uuid: "f83afa4c3fa644f08196bf2ef04d733b" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "f83afa4c3fa644f08196bf2ef04d733b" member_type: VOTER last_known_addr { host: "127.1.170.190" port: 42111 } } peers { permanent_uuid: "2fc4ff3aee844aa492e3b79e53ad59c3" member_type: NON_VOTER last_known_addr { host: "127.1.170.189" port: 40091 } attrs { promote: true } } }
I20260923 20:09:39.938881 8514 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2fc4ff3aee844aa492e3b79e53ad59c3 [sys.catalog]: This master's current role is: LEARNER
W20260923 20:09:39.957165 8439 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:09:39.957459 8439 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:09:39.957609 8439 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:09:39.965758 8439 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 20:09:39.965935 8439 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:09:39.966022 8439 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:09:39.966125 8439 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 20:09:39.966207 8439 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 20:09:39.976926 8439 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.170.148:44293
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194121577027-1706-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194121577027-1706-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.1.170.190:42111,127.1.170.189:40091,127.1.170.188:36059
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194121577027-1706-0/krb5kdc/oryx_127.1.170.188.keytab
--principal=oryx/127.1.170.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.170.188:36059
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194121577027-1706-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.1.170.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194121577027-1706-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:09:39.978426 8439 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:09:39.979732 8439 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:09:39.992374 8531 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:09:39.992583 8534 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:09:39.992777 8532 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:09:39.993755 8439 server_base.cc:1034] running on GCE node
I20260923 20:09:39.994642 8439 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:09:39.996188 8439 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:09:39.997423 8439 hybrid_clock.cc:648] HybridClock initialized: now 1790194179997374 us; error 39 us; skew 500 ppm
I20260923 20:09:39.997807 8439 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:09:39 dist-test-slave-th4g krb5kdc[7830](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194179, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.170.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:09:40.001317 8439 init.cc:377] Logged in from keytab as oryx/127.1.170.188@KRBTEST.COM (short username oryx)
I20260923 20:09:40.002950 8439 webserver.cc:460] Webserver started at http://127.1.170.188:34853/ using document root <none> and password file <none>
I20260923 20:09:40.003681 8439 fs_manager.cc:359] Metadata directory not provided
I20260923 20:09:40.003829 8439 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:09:40.004132 8439 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:09:40.007157 8439 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194121577027-1706-0/minicluster-data/master-2/data/instance:
uuid: "0a16f1acbd7c4420b8d8cdf2732a5200"
format_stamp: "Formatted at 2026-09-23 20:09:39 on dist-test-slave-th4g"
server_key: "1f7bacf65c36a43139b6f606f3633ad1"
server_key_iv: "dfae9738fae2da12feec54bc46d27aec"
server_key_version: "encryptionkey@0"
I20260923 20:09:40.007953 8439 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194121577027-1706-0/minicluster-data/master-2/wal/instance:
uuid: "0a16f1acbd7c4420b8d8cdf2732a5200"
format_stamp: "Formatted at 2026-09-23 20:09:39 on dist-test-slave-th4g"
server_key: "1f7bacf65c36a43139b6f606f3633ad1"
server_key_iv: "dfae9738fae2da12feec54bc46d27aec"
server_key_version: "encryptionkey@0"
I20260923 20:09:40.013353 8439 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260923 20:09:40.017302 8541 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:09:40.018846 8439 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 20:09:40.019058 8439 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194121577027-1706-0/minicluster-data/master-2/data,/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194121577027-1706-0/minicluster-data/master-2/wal
uuid: "0a16f1acbd7c4420b8d8cdf2732a5200"
format_stamp: "Formatted at 2026-09-23 20:09:39 on dist-test-slave-th4g"
server_key: "1f7bacf65c36a43139b6f606f3633ad1"
server_key_iv: "dfae9738fae2da12feec54bc46d27aec"
server_key_version: "encryptionkey@0"
I20260923 20:09:40.019364 8439 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194121577027-1706-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194121577027-1706-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194121577027-1706-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:09:40.048355 8439 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:09:40.052917 8439 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:09:40.053233 8439 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:09:40.072816 8439 rpc_server.cc:307] RPC server started. Bound to: 127.1.170.188:36059
I20260923 20:09:40.072837 8592 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.170.188:36059 every 8 connection(s)
I20260923 20:09:40.074368 8439 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194121577027-1706-0/minicluster-data/master-2/data/info.pb
I20260923 20:09:40.078120 8593 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:09:40.083074 1706 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqUuku6/build/asan/bin/kudu as pid 8439
I20260923 20:09:40.085021 8593 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.170.190" port: 42111 } has no permanent_uuid. Determining permanent_uuid...
Sep 23 20:09:40 dist-test-slave-th4g krb5kdc[7830](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194179, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.170.188@KRBTEST.COM for oryx/127.1.170.190@KRBTEST.COM
Sep 23 20:09:40 dist-test-slave-th4g krb5kdc[7830](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194176, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.1.170.188@KRBTEST.COM
I20260923 20:09:40.106168 8410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:09:40.088667 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41864 (local address 127.1.170.190:42111)
0923 20:09:40.088895 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0923 20:09:40.088905 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0923 20:09:40.089579 (+ 674us) server_negotiation.cc:407] Connection header received
0923 20:09:40.089758 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:09:40.089763 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:09:40.089842 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:09:40.089974 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:09:40.091784 (+ 1810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:09:40.092461 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:09:40.094668 (+ 2207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:09:40.095036 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:09:40.095175 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:09:40.098807 (+ 3632us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:09:40.098833 (+ 26us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:09:40.098838 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:09:40.098880 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:09:40.102116 (+ 3236us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:09:40.103113 (+ 997us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:09:40.103117 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:09:40.103119 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:09:40.103244 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:09:40.103707 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:09:40.103712 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:09:40.103716 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:09:40.104066 (+ 350us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:09:40.104172 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0923 20:09:40.105520 (+ 1348us) server_negotiation.cc:299] Negotiation successful
0923 20:09:40.105736 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260923 20:09:40.106956 8597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:09:40.088597 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.190:42111 (local address 127.0.0.1:41864)
0923 20:09:40.089424 (+ 827us) negotiation.cc:107] Waiting for socket to connect
0923 20:09:40.089473 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0923 20:09:40.089641 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:09:40.090213 (+ 572us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:09:40.090221 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:09:40.091091 (+ 870us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:09:40.091617 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:09:40.091634 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:09:40.092602 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:09:40.092608 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:09:40.094520 (+ 1912us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:09:40.094534 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:09:40.095268 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:09:40.095274 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:09:40.095432 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:09:40.096087 (+ 655us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:09:40.096113 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:09:40.098601 (+ 2488us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:09:40.102238 (+ 3637us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:09:40.102247 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:09:40.102261 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:09:40.102967 (+ 706us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:09:40.103427 (+ 460us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:09:40.103430 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:09:40.103432 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:09:40.103593 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:09:40.104229 (+ 636us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:09:40.104237 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:09:40.104626 (+ 389us) client_negotiation.cc:769] Sending connection context
0923 20:09:40.105014 (+ 388us) client_negotiation.cc:240] Negotiation successful
0923 20:09:40.105547 (+ 533us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":586,"spinlock_wait_cycles":21376,"thread_start_us":372,"threads_started":1}
I20260923 20:09:40.106974 8596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:09:40.085230 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41080 (local address 127.1.170.188:36059)
0923 20:09:40.086428 (+ 1198us) server_negotiation.cc:206] Beginning negotiation
0923 20:09:40.086443 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0923 20:09:40.086494 (+ 51us) server_negotiation.cc:407] Connection header received
0923 20:09:40.088650 (+ 2156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:09:40.088677 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:09:40.089266 (+ 589us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:09:40.089669 (+ 403us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:09:40.091588 (+ 1919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:09:40.092752 (+ 1164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:09:40.094538 (+ 1786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:09:40.095032 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:09:40.095233 (+ 201us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:09:40.099862 (+ 4629us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:09:40.099875 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:09:40.099893 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:09:40.099930 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:09:40.103214 (+ 3284us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:09:40.103973 (+ 759us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:09:40.103979 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:09:40.103981 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:09:40.104094 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:09:40.104550 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:09:40.104555 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:09:40.104558 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:09:40.104942 (+ 384us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:09:40.105211 (+ 269us) server_negotiation.cc:1035] Waiting for connection context
0923 20:09:40.105532 (+ 321us) server_negotiation.cc:299] Negotiation successful
0923 20:09:40.105739 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":939,"thread_start_us":727,"threads_started":1}
I20260923 20:09:40.109248 8593 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.170.189" port: 40091 } has no permanent_uuid. Determining permanent_uuid...
Sep 23 20:09:40 dist-test-slave-th4g krb5kdc[7830](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194179, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.170.188@KRBTEST.COM for oryx/127.1.170.189@KRBTEST.COM
I20260923 20:09:40.122184 8597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:09:40.110061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.189:40091 (local address 127.0.0.1:45360)
0923 20:09:40.110227 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0923 20:09:40.110245 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0923 20:09:40.110354 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:09:40.110664 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:09:40.110669 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:09:40.111020 (+ 351us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:09:40.111142 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:09:40.111150 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:09:40.112120 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:09:40.112126 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:09:40.113032 (+ 906us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:09:40.113042 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:09:40.113766 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:09:40.113772 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:09:40.113961 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:09:40.114534 (+ 573us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:09:40.114551 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:09:40.116833 (+ 2282us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:09:40.119827 (+ 2994us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:09:40.119834 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:09:40.119838 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:09:40.120207 (+ 369us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:09:40.120585 (+ 378us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:09:40.120589 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:09:40.120593 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:09:40.120719 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:09:40.121229 (+ 510us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:09:40.121234 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:09:40.121373 (+ 139us) client_negotiation.cc:769] Sending connection context
0923 20:09:40.121510 (+ 137us) client_negotiation.cc:240] Negotiation successful
0923 20:09:40.121806 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":54}
I20260923 20:09:40.122184 8526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:09:40.110014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45360 (local address 127.1.170.189:40091)
0923 20:09:40.110232 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0923 20:09:40.110240 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 20:09:40.110293 (+ 53us) server_negotiation.cc:407] Connection header received
0923 20:09:40.110469 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:09:40.110474 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:09:40.110548 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:09:40.110649 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:09:40.111291 (+ 642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:09:40.111984 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:09:40.113186 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:09:40.113602 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:09:40.113738 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:09:40.117026 (+ 3288us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:09:40.117031 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:09:40.117034 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:09:40.117064 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:09:40.119672 (+ 2608us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:09:40.120333 (+ 661us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:09:40.120345 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:09:40.120348 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:09:40.120468 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:09:40.120827 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:09:40.120833 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:09:40.120837 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:09:40.121109 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:09:40.121260 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0923 20:09:40.121592 (+ 332us) server_negotiation.cc:299] Negotiation successful
0923 20:09:40.121833 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260923 20:09:40.123453 8593 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.170.188" port: 36059 } has no permanent_uuid. Determining permanent_uuid...
Sep 23 20:09:40 dist-test-slave-th4g krb5kdc[7830](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194179, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.170.188@KRBTEST.COM for oryx/127.1.170.188@KRBTEST.COM
I20260923 20:09:40.135937 8597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:09:40.124269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.188:36059 (local address 127.0.0.1:41092)
0923 20:09:40.124467 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0923 20:09:40.124487 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0923 20:09:40.124585 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:09:40.124845 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:09:40.124849 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:09:40.125147 (+ 298us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:09:40.125325 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:09:40.125333 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:09:40.126371 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:09:40.126375 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:09:40.127166 (+ 791us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:09:40.127176 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:09:40.127860 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:09:40.127864 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:09:40.128017 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:09:40.128588 (+ 571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:09:40.128604 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:09:40.130643 (+ 2039us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:09:40.133477 (+ 2834us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:09:40.133480 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:09:40.133483 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:09:40.133840 (+ 357us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:09:40.134260 (+ 420us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:09:40.134264 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:09:40.134266 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:09:40.134476 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:09:40.135002 (+ 526us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:09:40.135006 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:09:40.135120 (+ 114us) client_negotiation.cc:769] Sending connection context
0923 20:09:40.135276 (+ 156us) client_negotiation.cc:240] Negotiation successful
0923 20:09:40.135525 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":65}
I20260923 20:09:40.135959 8596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:09:40.124436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41092 (local address 127.1.170.188:36059)
0923 20:09:40.124585 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0923 20:09:40.124589 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 20:09:40.124604 (+ 15us) server_negotiation.cc:407] Connection header received
0923 20:09:40.124685 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:09:40.124688 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:09:40.124738 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:09:40.124812 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:09:40.125454 (+ 642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:09:40.126233 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:09:40.127305 (+ 1072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:09:40.127699 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:09:40.127795 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:09:40.130829 (+ 3034us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:09:40.130834 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:09:40.130837 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:09:40.130864 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:09:40.133346 (+ 2482us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:09:40.134005 (+ 659us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:09:40.134008 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:09:40.134011 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:09:40.134118 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:09:40.134576 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:09:40.134579 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:09:40.134581 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:09:40.134898 (+ 317us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:09:40.135003 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0923 20:09:40.135351 (+ 348us) server_negotiation.cc:299] Negotiation successful
0923 20:09:40.135632 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":66,"server-negotiator.queue_time_us":54}
I20260923 20:09:40.140882 8593 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0a16f1acbd7c4420b8d8cdf2732a5200: Bootstrap starting.
I20260923 20:09:40.144384 8593 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0a16f1acbd7c4420b8d8cdf2732a5200: Neither blocks nor log segments found. Creating new log.
I20260923 20:09:40.145452 8593 log.cc:824] T 00000000000000000000000000000000 P 0a16f1acbd7c4420b8d8cdf2732a5200: Log is configured to *not* fsync() on all Append() calls
I20260923 20:09:40.148102 8593 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0a16f1acbd7c4420b8d8cdf2732a5200: No bootstrap required, opened a new log
I20260923 20:09:40.155541 8593 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0a16f1acbd7c4420b8d8cdf2732a5200 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f83afa4c3fa644f08196bf2ef04d733b" member_type: VOTER last_known_addr { host: "127.1.170.190" port: 42111 } } peers { permanent_uuid: "2fc4ff3aee844aa492e3b79e53ad59c3" member_type: VOTER last_known_addr { host: "127.1.170.189" port: 40091 } } peers { permanent_uuid: "0a16f1acbd7c4420b8d8cdf2732a5200" member_type: VOTER last_known_addr { host: "127.1.170.188" port: 36059 } }
I20260923 20:09:40.155833 8593 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0a16f1acbd7c4420b8d8cdf2732a5200 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:09:40.155941 8593 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0a16f1acbd7c4420b8d8cdf2732a5200 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a16f1acbd7c4420b8d8cdf2732a5200, State: Initialized, Role: FOLLOWER
I20260923 20:09:40.156531 8593 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0a16f1acbd7c4420b8d8cdf2732a5200 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f83afa4c3fa644f08196bf2ef04d733b" member_type: VOTER last_known_addr { host: "127.1.170.190" port: 42111 } } peers { permanent_uuid: "2fc4ff3aee844aa492e3b79e53ad59c3" member_type: VOTER last_known_addr { host: "127.1.170.189" port: 40091 } } peers { permanent_uuid: "0a16f1acbd7c4420b8d8cdf2732a5200" member_type: VOTER last_known_addr { host: "127.1.170.188" port: 36059 } }
I20260923 20:09:40.157388 8599 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0a16f1acbd7c4420b8d8cdf2732a5200 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f83afa4c3fa644f08196bf2ef04d733b" member_type: VOTER last_known_addr { host: "127.1.170.190" port: 42111 } } peers { permanent_uuid: "2fc4ff3aee844aa492e3b79e53ad59c3" member_type: VOTER last_known_addr { host: "127.1.170.189" port: 40091 } } peers { permanent_uuid: "0a16f1acbd7c4420b8d8cdf2732a5200" member_type: VOTER last_known_addr { host: "127.1.170.188" port: 36059 } } }
I20260923 20:09:40.157828 8599 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0a16f1acbd7c4420b8d8cdf2732a5200 [sys.catalog]: This master's current role is: FOLLOWER
I20260923 20:09:40.158449 8593 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0a16f1acbd7c4420b8d8cdf2732a5200 [sys.catalog]: configured and running, proceeding with master startup.
W20260923 20:09:40.167830 8610 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 0a16f1acbd7c4420b8d8cdf2732a5200: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260923 20:09:40.167974 8610 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260923 20:09:40.172063 8439 master_runner.cc:428] Detected that this master 0a16f1acbd7c4420b8d8cdf2732a5200 is joining an existing cluster
I20260923 20:09:40.172230 8439 master_runner.cc:432] Initiating AddMaster RPC to add 127.1.170.188:36059
I20260923 20:09:40.198167 8526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:09:40.176600 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45362 (local address 127.1.170.189:40091)
0923 20:09:40.176794 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0923 20:09:40.176801 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:09:40.181929 (+ 5128us) server_negotiation.cc:407] Connection header received
0923 20:09:40.182493 (+ 564us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:09:40.182502 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:09:40.182603 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:09:40.182686 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:09:40.184410 (+ 1724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:09:40.185438 (+ 1028us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:09:40.188009 (+ 2571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:09:40.188697 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:09:40.188792 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:09:40.191688 (+ 2896us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:09:40.191694 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:09:40.191699 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:09:40.191736 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:09:40.195293 (+ 3557us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:09:40.196059 (+ 766us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:09:40.196064 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:09:40.196068 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:09:40.196205 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:09:40.196618 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:09:40.196623 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:09:40.196627 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:09:40.196919 (+ 292us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:09:40.197034 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0923 20:09:40.197585 (+ 551us) server_negotiation.cc:299] Negotiation successful
0923 20:09:40.197817 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260923 20:09:40.198681 8596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:09:40.176853 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41108 (local address 127.1.170.188:36059)
0923 20:09:40.177201 (+ 348us) server_negotiation.cc:206] Beginning negotiation
0923 20:09:40.177209 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 20:09:40.180665 (+ 3456us) server_negotiation.cc:407] Connection header received
0923 20:09:40.180865 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:09:40.180872 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:09:40.180946 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:09:40.181048 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:09:40.182742 (+ 1694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:09:40.183915 (+ 1173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:09:40.187226 (+ 3311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:09:40.188040 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:09:40.188133 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:09:40.191446 (+ 3313us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:09:40.191452 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:09:40.191457 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:09:40.191496 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:09:40.195748 (+ 4252us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:09:40.196481 (+ 733us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:09:40.196488 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:09:40.196493 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:09:40.196640 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:09:40.197069 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:09:40.197074 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:09:40.197076 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:09:40.197439 (+ 363us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:09:40.197577 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0923 20:09:40.198105 (+ 528us) server_negotiation.cc:299] Negotiation successful
0923 20:09:40.198312 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":6,"server-negotiator.queue_time_us":215}
I20260923 20:09:40.199819 8616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:09:40.176726 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.188:36059 (local address 127.0.0.1:41108)
0923 20:09:40.180506 (+ 3780us) negotiation.cc:107] Waiting for socket to connect
0923 20:09:40.180531 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0923 20:09:40.180723 (+ 192us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:09:40.181271 (+ 548us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:09:40.181278 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:09:40.182113 (+ 835us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:09:40.182580 (+ 467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:09:40.182594 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:09:40.184201 (+ 1607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:09:40.184206 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:09:40.187073 (+ 2867us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:09:40.187087 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:09:40.188497 (+ 1410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:09:40.188507 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:09:40.188696 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:09:40.189890 (+ 1194us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:09:40.189914 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:09:40.191261 (+ 1347us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:09:40.195878 (+ 4617us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:09:40.195883 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:09:40.195886 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:09:40.196355 (+ 469us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:09:40.196777 (+ 422us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:09:40.196783 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:09:40.196787 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:09:40.196948 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:09:40.197649 (+ 701us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:09:40.197655 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:09:40.197818 (+ 163us) client_negotiation.cc:769] Sending connection context
0923 20:09:40.198920 (+ 1102us) client_negotiation.cc:240] Negotiation successful
0923 20:09:40.199313 (+ 393us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3592,"mutex_wait_us":106,"thread_start_us":3398,"threads_started":1}
I20260923 20:09:40.200291 8617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:09:40.177104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.189:40091 (local address 127.0.0.1:45362)
0923 20:09:40.181789 (+ 4685us) negotiation.cc:107] Waiting for socket to connect
0923 20:09:40.181809 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0923 20:09:40.182288 (+ 479us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:09:40.182932 (+ 644us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:09:40.182939 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:09:40.183786 (+ 847us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:09:40.184210 (+ 424us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:09:40.184224 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:09:40.185647 (+ 1423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:09:40.185657 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:09:40.187859 (+ 2202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:09:40.187871 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:09:40.189351 (+ 1480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:09:40.189362 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:09:40.189611 (+ 249us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:09:40.190304 (+ 693us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:09:40.190332 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:09:40.191472 (+ 1140us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:09:40.195447 (+ 3975us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:09:40.195451 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:09:40.195456 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:09:40.195939 (+ 483us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:09:40.196343 (+ 404us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:09:40.196347 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:09:40.196351 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:09:40.196512 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:09:40.197103 (+ 591us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:09:40.197107 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:09:40.197324 (+ 217us) client_negotiation.cc:769] Sending connection context
0923 20:09:40.198694 (+ 1370us) client_negotiation.cc:240] Negotiation successful
0923 20:09:40.199506 (+ 812us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4520,"mutex_wait_us":230,"thread_start_us":4848,"threads_started":1}
I20260923 20:09:40.204497 8410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:09:40.176665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41872 (local address 127.1.170.190:42111)
0923 20:09:40.177046 (+ 381us) server_negotiation.cc:206] Beginning negotiation
0923 20:09:40.177052 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:09:40.184498 (+ 7446us) server_negotiation.cc:407] Connection header received
0923 20:09:40.184922 (+ 424us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:09:40.184928 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:09:40.185002 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:09:40.185066 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:09:40.187051 (+ 1985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:09:40.188106 (+ 1055us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:09:40.195011 (+ 6905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:09:40.195748 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:09:40.195855 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:09:40.198241 (+ 2386us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:09:40.198249 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:09:40.198254 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:09:40.198296 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:09:40.201839 (+ 3543us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:09:40.202471 (+ 632us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:09:40.202476 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:09:40.202480 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:09:40.202620 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:09:40.203004 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:09:40.203009 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:09:40.203012 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:09:40.203339 (+ 327us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:09:40.203527 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0923 20:09:40.203922 (+ 395us) server_negotiation.cc:299] Negotiation successful
0923 20:09:40.204129 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":253}
I20260923 20:09:40.204567 8618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:09:40.177585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.190:42111 (local address 127.0.0.1:41872)
0923 20:09:40.184396 (+ 6811us) negotiation.cc:107] Waiting for socket to connect
0923 20:09:40.184419 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0923 20:09:40.184779 (+ 360us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:09:40.185253 (+ 474us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:09:40.185258 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:09:40.186062 (+ 804us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:09:40.186873 (+ 811us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:09:40.186886 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:09:40.188246 (+ 1360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:09:40.188251 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:09:40.194826 (+ 6575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:09:40.194841 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:09:40.196169 (+ 1328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:09:40.196177 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:09:40.196369 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:09:40.196964 (+ 595us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:09:40.196988 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:09:40.198040 (+ 1052us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:09:40.202010 (+ 3970us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:09:40.202015 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:09:40.202018 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:09:40.202338 (+ 320us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:09:40.202746 (+ 408us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:09:40.202751 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:09:40.202754 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:09:40.202887 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:09:40.203457 (+ 570us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:09:40.203464 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:09:40.203668 (+ 204us) client_negotiation.cc:769] Sending connection context
0923 20:09:40.203812 (+ 144us) client_negotiation.cc:240] Negotiation successful
0923 20:09:40.204121 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6664,"mutex_wait_us":4,"thread_start_us":6927,"threads_started":1}
I20260923 20:09:40.213903 1706 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskqUuku6/build/asan/bin/kudu pid 7936
Sep 23 20:09:40 dist-test-slave-th4g krb5kdc[7830](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194177, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.170.190@KRBTEST.COM for oryx/127.1.170.188@KRBTEST.COM
I20260923 20:09:40.223888 8433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:09:40.209310 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.188:36059 (local address 127.0.0.1:41114)
0923 20:09:40.209469 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0923 20:09:40.209486 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0923 20:09:40.209576 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:09:40.210001 (+ 425us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:09:40.210010 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:09:40.210417 (+ 407us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:09:40.210562 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:09:40.210570 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:09:40.211583 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:09:40.211590 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:09:40.212692 (+ 1102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:09:40.212705 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:09:40.213846 (+ 1141us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:09:40.213850 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:09:40.214060 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:09:40.214659 (+ 599us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:09:40.214683 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:09:40.217428 (+ 2745us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:09:40.221265 (+ 3837us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:09:40.221273 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:09:40.221277 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:09:40.221618 (+ 341us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:09:40.222058 (+ 440us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:09:40.222063 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:09:40.222066 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:09:40.222192 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:09:40.222820 (+ 628us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:09:40.222826 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:09:40.222996 (+ 170us) client_negotiation.cc:769] Sending connection context
0923 20:09:40.223150 (+ 154us) client_negotiation.cc:240] Negotiation successful
0923 20:09:40.223476 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":56}
I20260923 20:09:40.224756 8596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:09:40.209339 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41114 (local address 127.1.170.188:36059)
0923 20:09:40.209499 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0923 20:09:40.209505 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:09:40.209529 (+ 24us) server_negotiation.cc:407] Connection header received
0923 20:09:40.209744 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:09:40.209750 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:09:40.209837 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:09:40.209972 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:09:40.210715 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:09:40.211452 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:09:40.212843 (+ 1391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:09:40.213552 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:09:40.213670 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:09:40.217590 (+ 3920us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:09:40.218006 (+ 416us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:09:40.218015 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:09:40.218057 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:09:40.221120 (+ 3063us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:09:40.221745 (+ 625us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:09:40.221752 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:09:40.221755 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:09:40.221942 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:09:40.222294 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:09:40.222301 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:09:40.222304 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:09:40.222668 (+ 364us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:09:40.222803 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0923 20:09:40.224174 (+ 1371us) server_negotiation.cc:299] Negotiation successful
0923 20:09:40.224390 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260923 20:09:40.226354 7875 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.1.170.188:36059: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "0a16f1acbd7c4420b8d8cdf2732a5200"
member_type: NON_VOTER
last_known_addr {
host: "127.1.170.188"
port: 36059
}
attrs {
promote: true
}
}
dest_uuid: "f83afa4c3fa644f08196bf2ef04d733b"
cas_config_opid_index: 5
Sep 23 20:09:40 dist-test-slave-th4g krb5kdc[7830](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194176, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.1.170.129@KRBTEST.COM
I20260923 20:09:40.227210 7875 consensus_queue.cc:237] T 00000000000000000000000000000000 P f83afa4c3fa644f08196bf2ef04d733b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "f83afa4c3fa644f08196bf2ef04d733b" member_type: VOTER last_known_addr { host: "127.1.170.190" port: 42111 } } peers { permanent_uuid: "2fc4ff3aee844aa492e3b79e53ad59c3" member_type: NON_VOTER last_known_addr { host: "127.1.170.189" port: 40091 } attrs { promote: true } } peers { permanent_uuid: "0a16f1acbd7c4420b8d8cdf2732a5200" member_type: NON_VOTER last_known_addr { host: "127.1.170.188" port: 36059 } attrs { promote: true } }
I20260923 20:09:40.229020 8436 raft_consensus.cc:2932] T 00000000000000000000000000000000 P f83afa4c3fa644f08196bf2ef04d733b [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 0a16f1acbd7c4420b8d8cdf2732a5200 (127.1.170.188) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "f83afa4c3fa644f08196bf2ef04d733b" member_type: VOTER last_known_addr { host: "127.1.170.190" port: 42111 } } peers { permanent_uuid: "2fc4ff3aee844aa492e3b79e53ad59c3" member_type: NON_VOTER last_known_addr { host: "127.1.170.189" port: 40091 } attrs { promote: true } } peers { permanent_uuid: "0a16f1acbd7c4420b8d8cdf2732a5200" member_type: NON_VOTER last_known_addr { host: "127.1.170.188" port: 36059 } attrs { promote: true } } }
I20260923 20:09:40.230077 8568 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0a16f1acbd7c4420b8d8cdf2732a5200 [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:09:40.234182 8568 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 0a16f1acbd7c4420b8d8cdf2732a5200 [term 1 FOLLOWER]: Refusing update from remote peer f83afa4c3fa644f08196bf2ef04d733b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260923 20:09:40.236316 8486 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 2fc4ff3aee844aa492e3b79e53ad59c3 [term 1 LEARNER]: Refusing update from remote peer f83afa4c3fa644f08196bf2ef04d733b: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260923 20:09:40.236828 8619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:09:40.214992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48574 (local address 127.1.170.129:41229)
0923 20:09:40.215607 (+ 615us) server_negotiation.cc:206] Beginning negotiation
0923 20:09:40.215617 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0923 20:09:40.215647 (+ 30us) server_negotiation.cc:407] Connection header received
0923 20:09:40.215724 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:09:40.215736 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:09:40.216029 (+ 293us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:09:40.216207 (+ 178us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:09:40.217497 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:09:40.219526 (+ 2029us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:09:40.221381 (+ 1855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:09:40.222292 (+ 911us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:09:40.222422 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:09:40.227778 (+ 5356us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:09:40.227792 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:09:40.227811 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:09:40.227849 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:09:40.231866 (+ 4017us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:09:40.232750 (+ 884us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:09:40.232759 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:09:40.232762 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:09:40.232914 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:09:40.233419 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:09:40.233427 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:09:40.233433 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:09:40.233993 (+ 560us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:09:40.235647 (+ 1654us) server_negotiation.cc:1035] Waiting for connection context
0923 20:09:40.235845 (+ 198us) server_negotiation.cc:299] Negotiation successful
0923 20:09:40.236156 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":437,"thread_start_us":282,"threads_started":1}
I20260923 20:09:40.238620 8059 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:09:40.242161 8436 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.1.170.188:36059
I20260923 20:09:40.242700 8620 consensus_queue.cc:1035] T 00000000000000000000000000000000 P f83afa4c3fa644f08196bf2ef04d733b [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a16f1acbd7c4420b8d8cdf2732a5200" member_type: NON_VOTER last_known_addr { host: "127.1.170.188" port: 36059 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 20:09:40.243628 8439 master.cc:537] Master@127.1.170.188:36059 shutting down...
I20260923 20:09:40.242928 8621 sys_catalog.cc:455] T 00000000000000000000000000000000 P f83afa4c3fa644f08196bf2ef04d733b [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "f83afa4c3fa644f08196bf2ef04d733b" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "f83afa4c3fa644f08196bf2ef04d733b" member_type: VOTER last_known_addr { host: "127.1.170.190" port: 42111 } } peers { permanent_uuid: "2fc4ff3aee844aa492e3b79e53ad59c3" member_type: NON_VOTER last_known_addr { host: "127.1.170.189" port: 40091 } attrs { promote: true } } peers { permanent_uuid: "0a16f1acbd7c4420b8d8cdf2732a5200" member_type: NON_VOTER last_known_addr { host: "127.1.170.188" port: 36059 } attrs { promote: true } } }
I20260923 20:09:40.246343 8621 sys_catalog.cc:458] T 00000000000000000000000000000000 P f83afa4c3fa644f08196bf2ef04d733b [sys.catalog]: This master's current role is: LEADER
I20260923 20:09:40.246771 8436 consensus_queue.cc:1035] T 00000000000000000000000000000000 P f83afa4c3fa644f08196bf2ef04d733b [LEADER]: Connected to new peer: Peer: permanent_uuid: "2fc4ff3aee844aa492e3b79e53ad59c3" member_type: NON_VOTER last_known_addr { host: "127.1.170.189" port: 40091 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260923 20:09:40.266207 8486 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 2fc4ff3aee844aa492e3b79e53ad59c3 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 0a16f1acbd7c4420b8d8cdf2732a5200 (127.1.170.188) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "f83afa4c3fa644f08196bf2ef04d733b" member_type: VOTER last_known_addr { host: "127.1.170.190" port: 42111 } } peers { permanent_uuid: "2fc4ff3aee844aa492e3b79e53ad59c3" member_type: NON_VOTER last_known_addr { host: "127.1.170.189" port: 40091 } attrs { promote: true } } peers { permanent_uuid: "0a16f1acbd7c4420b8d8cdf2732a5200" member_type: NON_VOTER last_known_addr { host: "127.1.170.188" port: 36059 } attrs { promote: true } } }
I20260923 20:09:40.270998 8514 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2fc4ff3aee844aa492e3b79e53ad59c3 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f83afa4c3fa644f08196bf2ef04d733b" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "f83afa4c3fa644f08196bf2ef04d733b" member_type: VOTER last_known_addr { host: "127.1.170.190" port: 42111 } } peers { permanent_uuid: "2fc4ff3aee844aa492e3b79e53ad59c3" member_type: NON_VOTER last_known_addr { host: "127.1.170.189" port: 40091 } attrs { promote: true } } peers { permanent_uuid: "0a16f1acbd7c4420b8d8cdf2732a5200" member_type: NON_VOTER last_known_addr { host: "127.1.170.188" port: 36059 } attrs { promote: true } } }
I20260923 20:09:40.271287 8514 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2fc4ff3aee844aa492e3b79e53ad59c3 [sys.catalog]: This master's current role is: LEARNER
I20260923 20:09:40.272478 8620 raft_consensus.cc:1050] T 00000000000000000000000000000000 P f83afa4c3fa644f08196bf2ef04d733b: attempting to promote NON_VOTER 2fc4ff3aee844aa492e3b79e53ad59c3 to VOTER
I20260923 20:09:40.273384 8620 consensus_queue.cc:237] T 00000000000000000000000000000000 P f83afa4c3fa644f08196bf2ef04d733b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f83afa4c3fa644f08196bf2ef04d733b" member_type: VOTER last_known_addr { host: "127.1.170.190" port: 42111 } } peers { permanent_uuid: "2fc4ff3aee844aa492e3b79e53ad59c3" member_type: VOTER last_known_addr { host: "127.1.170.189" port: 40091 } attrs { promote: false } } peers { permanent_uuid: "0a16f1acbd7c4420b8d8cdf2732a5200" member_type: NON_VOTER last_known_addr { host: "127.1.170.188" port: 36059 } attrs { promote: true } }
I20260923 20:09:40.276055 8486 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 2fc4ff3aee844aa492e3b79e53ad59c3 [term 1 LEARNER]: Refusing update from remote peer f83afa4c3fa644f08196bf2ef04d733b: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260923 20:09:40.276573 8622 consensus_queue.cc:1035] T 00000000000000000000000000000000 P f83afa4c3fa644f08196bf2ef04d733b [LEADER]: Connected to new peer: Peer: permanent_uuid: "2fc4ff3aee844aa492e3b79e53ad59c3" member_type: VOTER last_known_addr { host: "127.1.170.189" port: 40091 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260923 20:09:40.279018 8436 raft_consensus.cc:2932] T 00000000000000000000000000000000 P f83afa4c3fa644f08196bf2ef04d733b [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 2fc4ff3aee844aa492e3b79e53ad59c3 (127.1.170.189) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f83afa4c3fa644f08196bf2ef04d733b" member_type: VOTER last_known_addr { host: "127.1.170.190" port: 42111 } } peers { permanent_uuid: "2fc4ff3aee844aa492e3b79e53ad59c3" member_type: VOTER last_known_addr { host: "127.1.170.189" port: 40091 } attrs { promote: false } } peers { permanent_uuid: "0a16f1acbd7c4420b8d8cdf2732a5200" member_type: NON_VOTER last_known_addr { host: "127.1.170.188" port: 36059 } attrs { promote: true } } }
W20260923 20:09:40.283015 7861 consensus_peers.cc:487] T 00000000000000000000000000000000 P f83afa4c3fa644f08196bf2ef04d733b -> Peer 0a16f1acbd7c4420b8d8cdf2732a5200 (127.1.170.188:36059): Couldn't send request to peer 0a16f1acbd7c4420b8d8cdf2732a5200. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260923 20:09:40.283799 8485 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 2fc4ff3aee844aa492e3b79e53ad59c3 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 2fc4ff3aee844aa492e3b79e53ad59c3 (127.1.170.189) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f83afa4c3fa644f08196bf2ef04d733b" member_type: VOTER last_known_addr { host: "127.1.170.190" port: 42111 } } peers { permanent_uuid: "2fc4ff3aee844aa492e3b79e53ad59c3" member_type: VOTER last_known_addr { host: "127.1.170.189" port: 40091 } attrs { promote: false } } peers { permanent_uuid: "0a16f1acbd7c4420b8d8cdf2732a5200" member_type: NON_VOTER last_known_addr { host: "127.1.170.188" port: 36059 } attrs { promote: true } } }
I20260923 20:09:40.288622 8439 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 0a16f1acbd7c4420b8d8cdf2732a5200 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 20:09:40.289045 8439 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 0a16f1acbd7c4420b8d8cdf2732a5200 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 20:09:40.288411 8622 sys_catalog.cc:455] T 00000000000000000000000000000000 P f83afa4c3fa644f08196bf2ef04d733b [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "f83afa4c3fa644f08196bf2ef04d733b" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f83afa4c3fa644f08196bf2ef04d733b" member_type: VOTER last_known_addr { host: "127.1.170.190" port: 42111 } } peers { permanent_uuid: "2fc4ff3aee844aa492e3b79e53ad59c3" member_type: VOTER last_known_addr { host: "127.1.170.189" port: 40091 } attrs { promote: false } } peers { permanent_uuid: "0a16f1acbd7c4420b8d8cdf2732a5200" member_type: NON_VOTER last_known_addr { host: "127.1.170.188" port: 36059 } attrs { promote: true } } }
I20260923 20:09:40.289252 8439 tablet_replica.cc:330] T 00000000000000000000000000000000 P 0a16f1acbd7c4420b8d8cdf2732a5200: stopping tablet replica
I20260923 20:09:40.289273 8622 sys_catalog.cc:458] T 00000000000000000000000000000000 P f83afa4c3fa644f08196bf2ef04d733b [sys.catalog]: This master's current role is: LEADER
I20260923 20:09:40.288940 8436 sys_catalog.cc:455] T 00000000000000000000000000000000 P f83afa4c3fa644f08196bf2ef04d733b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f83afa4c3fa644f08196bf2ef04d733b" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f83afa4c3fa644f08196bf2ef04d733b" member_type: VOTER last_known_addr { host: "127.1.170.190" port: 42111 } } peers { permanent_uuid: "2fc4ff3aee844aa492e3b79e53ad59c3" member_type: VOTER last_known_addr { host: "127.1.170.189" port: 40091 } attrs { promote: false } } peers { permanent_uuid: "0a16f1acbd7c4420b8d8cdf2732a5200" member_type: NON_VOTER last_known_addr { host: "127.1.170.188" port: 36059 } attrs { promote: true } } }
I20260923 20:09:40.289690 8436 sys_catalog.cc:458] T 00000000000000000000000000000000 P f83afa4c3fa644f08196bf2ef04d733b [sys.catalog]: This master's current role is: LEADER
I20260923 20:09:40.291940 8514 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2fc4ff3aee844aa492e3b79e53ad59c3 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f83afa4c3fa644f08196bf2ef04d733b" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f83afa4c3fa644f08196bf2ef04d733b" member_type: VOTER last_known_addr { host: "127.1.170.190" port: 42111 } } peers { permanent_uuid: "2fc4ff3aee844aa492e3b79e53ad59c3" member_type: VOTER last_known_addr { host: "127.1.170.189" port: 40091 } attrs { promote: false } } peers { permanent_uuid: "0a16f1acbd7c4420b8d8cdf2732a5200" member_type: NON_VOTER last_known_addr { host: "127.1.170.188" port: 36059 } attrs { promote: true } } }
I20260923 20:09:40.292183 8514 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2fc4ff3aee844aa492e3b79e53ad59c3 [sys.catalog]: This master's current role is: FOLLOWER
I20260923 20:09:40.295994 8439 master.cc:552] Master@127.1.170.188:36059 shutdown complete.
I20260923 20:09:40.296082 8439 master_runner.cc:305] Clearing existing system tablet
I20260923 20:09:40.296455 8439 fs_manager.cc:359] Metadata directory not provided
I20260923 20:09:40.296558 8439 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:09:40.299414 8439 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.004s sys 0.000s
I20260923 20:09:40.301017 8629 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:09:40.301319 8439 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194121577027-1706-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:09:40.301725 8439 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260923 20:09:40.301898 8439 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194121577027-1706-0/minicluster-data/master-2/data,/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194121577027-1706-0/minicluster-data/master-2/wal
uuid: "0a16f1acbd7c4420b8d8cdf2732a5200"
format_stamp: "Formatted at 2026-09-23 20:09:39 on dist-test-slave-th4g"
server_key: "1f7bacf65c36a43139b6f606f3633ad1"
server_key_iv: "dfae9738fae2da12feec54bc46d27aec"
server_key_version: "encryptionkey@0"
I20260923 20:09:40.306885 8439 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 0a16f1acbd7c4420b8d8cdf2732a5200: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 20:09:40.313939 8439 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 0a16f1acbd7c4420b8d8cdf2732a5200: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260923 20:09:40.314190 8439 log.cc:1196] T 00000000000000000000000000000000 P 0a16f1acbd7c4420b8d8cdf2732a5200: Deleting WAL directory at /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194121577027-1706-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260923 20:09:40.314731 8439 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 0a16f1acbd7c4420b8d8cdf2732a5200: Deleting consensus metadata
I20260923 20:09:40.315017 8439 master_runner.cc:315] Copying system tablet from 127.1.170.190:42111
I20260923 20:09:40.317821 8439 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 0a16f1acbd7c4420b8d8cdf2732a5200: tablet copy: Beginning tablet copy session from remote peer at address 127.1.170.190:42111
I20260923 20:09:40.336916 8410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:09:40.318761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41874 (local address 127.1.170.190:42111)
0923 20:09:40.318970 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0923 20:09:40.318977 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:09:40.322524 (+ 3547us) server_negotiation.cc:407] Connection header received
0923 20:09:40.322705 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:09:40.322711 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:09:40.322797 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:09:40.322915 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:09:40.323878 (+ 963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:09:40.325004 (+ 1126us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:09:40.326919 (+ 1915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:09:40.327572 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:09:40.327721 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:09:40.330077 (+ 2356us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:09:40.330106 (+ 29us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:09:40.330111 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:09:40.330155 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:09:40.334014 (+ 3859us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:09:40.334793 (+ 779us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:09:40.334800 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:09:40.334805 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:09:40.334956 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:09:40.335386 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:09:40.335391 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:09:40.335395 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:09:40.335746 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:09:40.335961 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0923 20:09:40.336334 (+ 373us) server_negotiation.cc:299] Negotiation successful
0923 20:09:40.336550 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97}
I20260923 20:09:40.336993 8635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:09:40.318690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.190:42111 (local address 127.0.0.1:41874)
0923 20:09:40.322373 (+ 3683us) negotiation.cc:107] Waiting for socket to connect
0923 20:09:40.322398 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0923 20:09:40.322570 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:09:40.322959 (+ 389us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:09:40.322964 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:09:40.323474 (+ 510us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:09:40.323710 (+ 236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:09:40.323731 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:09:40.325168 (+ 1437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:09:40.325173 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:09:40.326748 (+ 1575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:09:40.326763 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:09:40.327765 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:09:40.327771 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:09:40.327966 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:09:40.328747 (+ 781us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:09:40.328770 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:09:40.329843 (+ 1073us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:09:40.334147 (+ 4304us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:09:40.334153 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:09:40.334158 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:09:40.334651 (+ 493us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:09:40.335065 (+ 414us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:09:40.335070 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:09:40.335074 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:09:40.335249 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:09:40.335882 (+ 633us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:09:40.335888 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:09:40.336073 (+ 185us) client_negotiation.cc:769] Sending connection context
0923 20:09:40.336237 (+ 164us) client_negotiation.cc:240] Negotiation successful
0923 20:09:40.336558 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3542,"thread_start_us":3387,"threads_started":1}
I20260923 20:09:40.337975 7895 tablet_copy_service.cc:140] P f83afa4c3fa644f08196bf2ef04d733b: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 0a16f1acbd7c4420b8d8cdf2732a5200 ({username='oryx', principal='oryx/127.1.170.188@KRBTEST.COM'} at 127.0.0.1:41874)
I20260923 20:09:40.338142 7895 tablet_copy_service.cc:161] P f83afa4c3fa644f08196bf2ef04d733b: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 0a16f1acbd7c4420b8d8cdf2732a5200 at {username='oryx', principal='oryx/127.1.170.188@KRBTEST.COM'} at 127.0.0.1:41874: session id = 0a16f1acbd7c4420b8d8cdf2732a5200-00000000000000000000000000000000
I20260923 20:09:40.341820 7895 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P f83afa4c3fa644f08196bf2ef04d733b: Tablet Copy: opened 0 blocks and 1 log segments
I20260923 20:09:40.343747 8439 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260923 20:09:40.351271 8439 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 0a16f1acbd7c4420b8d8cdf2732a5200: tablet copy: Starting download of 0 data blocks...
I20260923 20:09:40.351560 8439 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 0a16f1acbd7c4420b8d8cdf2732a5200: tablet copy: Starting download of 1 WAL segments...
I20260923 20:09:40.354059 8439 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 0a16f1acbd7c4420b8d8cdf2732a5200: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260923 20:09:40.358345 7895 tablet_copy_service.cc:342] P f83afa4c3fa644f08196bf2ef04d733b: Request end of tablet copy session 0a16f1acbd7c4420b8d8cdf2732a5200-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.1.170.188@KRBTEST.COM'} at 127.0.0.1:41874
I20260923 20:09:40.358498 7895 tablet_copy_service.cc:433] P f83afa4c3fa644f08196bf2ef04d733b: ending tablet copy session 0a16f1acbd7c4420b8d8cdf2732a5200-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 0a16f1acbd7c4420b8d8cdf2732a5200
I20260923 20:09:40.370427 8439 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:09:40.373222 8636 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:09:40.374158 8637 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:09:40.384555 8439 server_base.cc:1034] running on GCE node
W20260923 20:09:40.386176 8640 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:09:40.387038 8439 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:09:40.387259 1706 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqUuku6/build/asan/bin/kudu with pid 7936
I20260923 20:09:40.388403 8439 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:09:40.389606 8439 hybrid_clock.cc:648] HybridClock initialized: now 1790194180389555 us; error 42 us; skew 500 ppm
I20260923 20:09:40.390004 8439 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:09:40 dist-test-slave-th4g krb5kdc[7830](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194180, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.170.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:09:40.392458 8439 init.cc:377] Logged in from keytab as oryx/127.1.170.188@KRBTEST.COM (short username oryx)
I20260923 20:09:40.398976 8439 webserver.cc:460] Webserver started at http://127.1.170.188:44515/ using document root <none> and password file <none>
I20260923 20:09:40.399415 8439 fs_manager.cc:359] Metadata directory not provided
I20260923 20:09:40.399658 8439 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:09:40.403323 8439 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260923 20:09:40.405583 8646 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:09:40.406100 8439 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.002s sys 0.000s
I20260923 20:09:40.406281 8439 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194121577027-1706-0/minicluster-data/master-2/data,/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194121577027-1706-0/minicluster-data/master-2/wal
uuid: "0a16f1acbd7c4420b8d8cdf2732a5200"
format_stamp: "Formatted at 2026-09-23 20:09:39 on dist-test-slave-th4g"
server_key: "1f7bacf65c36a43139b6f606f3633ad1"
server_key_iv: "dfae9738fae2da12feec54bc46d27aec"
server_key_version: "encryptionkey@0"
I20260923 20:09:40.406553 8439 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194121577027-1706-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194121577027-1706-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194121577027-1706-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:09:40.413462 1706 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqUuku6/build/asan/bin/kudu
/tmp/dist-test-taskqUuku6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194121577027-1706-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194121577027-1706-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194121577027-1706-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194121577027-1706-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.170.129:41229
--local_ip_for_outbound_sockets=127.1.170.129
--tserver_master_addrs=127.1.170.190:42111
--webserver_port=34051
--webserver_interface=127.1.170.129
--keytab_file=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194121577027-1706-0/krb5kdc/oryx_127.1.170.129.keytab
--principal=oryx/127.1.170.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.170.148:44293
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.1.170.190:42111,127.1.170.189:40091
--tserver_master_addrs=127.1.170.190:42111,127.1.170.189:40091,127.1.170.188:36059 with env {KRB5CCNAME=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194121577027-1706-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194121577027-1706-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194121577027-1706-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260923 20:09:40.425064 8439 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:09:40.430058 8439 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:09:40.466881 8439 rpc_server.cc:307] RPC server started. Bound to: 127.1.170.188:36059
I20260923 20:09:40.466917 8698 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.170.188:36059 every 8 connection(s)
I20260923 20:09:40.467720 8439 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194121577027-1706-0/minicluster-data/master-2/data/info.pb
I20260923 20:09:40.475673 8699 sys_catalog.cc:263] Verifying existing consensus state
I20260923 20:09:40.478428 8699 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0a16f1acbd7c4420b8d8cdf2732a5200: Bootstrap starting.
I20260923 20:09:40.493273 8699 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0a16f1acbd7c4420b8d8cdf2732a5200: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260923 20:09:40.493769 8699 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0a16f1acbd7c4420b8d8cdf2732a5200: Bootstrap complete.
I20260923 20:09:40.494665 8699 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0a16f1acbd7c4420b8d8cdf2732a5200 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f83afa4c3fa644f08196bf2ef04d733b" member_type: VOTER last_known_addr { host: "127.1.170.190" port: 42111 } } peers { permanent_uuid: "2fc4ff3aee844aa492e3b79e53ad59c3" member_type: VOTER last_known_addr { host: "127.1.170.189" port: 40091 } attrs { promote: false } } peers { permanent_uuid: "0a16f1acbd7c4420b8d8cdf2732a5200" member_type: NON_VOTER last_known_addr { host: "127.1.170.188" port: 36059 } attrs { promote: true } }
I20260923 20:09:40.494916 8699 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0a16f1acbd7c4420b8d8cdf2732a5200 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 0a16f1acbd7c4420b8d8cdf2732a5200, State: Initialized, Role: LEARNER
I20260923 20:09:40.495151 8699 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0a16f1acbd7c4420b8d8cdf2732a5200 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f83afa4c3fa644f08196bf2ef04d733b" member_type: VOTER last_known_addr { host: "127.1.170.190" port: 42111 } } peers { permanent_uuid: "2fc4ff3aee844aa492e3b79e53ad59c3" member_type: VOTER last_known_addr { host: "127.1.170.189" port: 40091 } attrs { promote: false } } peers { permanent_uuid: "0a16f1acbd7c4420b8d8cdf2732a5200" member_type: NON_VOTER last_known_addr { host: "127.1.170.188" port: 36059 } attrs { promote: true } }
I20260923 20:09:40.495995 8702 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0a16f1acbd7c4420b8d8cdf2732a5200 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f83afa4c3fa644f08196bf2ef04d733b" member_type: VOTER last_known_addr { host: "127.1.170.190" port: 42111 } } peers { permanent_uuid: "2fc4ff3aee844aa492e3b79e53ad59c3" member_type: VOTER last_known_addr { host: "127.1.170.189" port: 40091 } attrs { promote: false } } peers { permanent_uuid: "0a16f1acbd7c4420b8d8cdf2732a5200" member_type: NON_VOTER last_known_addr { host: "127.1.170.188" port: 36059 } attrs { promote: true } } }
I20260923 20:09:40.496335 8699 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0a16f1acbd7c4420b8d8cdf2732a5200 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 20:09:40.497047 8702 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0a16f1acbd7c4420b8d8cdf2732a5200 [sys.catalog]: This master's current role is: LEARNER
I20260923 20:09:40.505692 8713 catalog_manager.cc:1259] Loaded cluster ID: 62f93adc2448407bb12c561910d92d81
I20260923 20:09:40.505831 8713 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 0a16f1acbd7c4420b8d8cdf2732a5200: loading cluster ID for follower catalog manager: success
I20260923 20:09:40.508515 8713 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 0a16f1acbd7c4420b8d8cdf2732a5200: acquiring CA information for follower catalog manager: success
I20260923 20:09:40.510330 8713 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 0a16f1acbd7c4420b8d8cdf2732a5200: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260923 20:09:40.801791 8651 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:09:40.802121 8651 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:09:40.802223 8651 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:09:40.809487 8651 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:09:40.809593 8651 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:09:40.809710 8651 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.170.129
I20260923 20:09:40.819849 8651 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.170.148:44293
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194121577027-1706-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194121577027-1706-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194121577027-1706-0/krb5kdc/oryx_127.1.170.129.keytab
--principal=oryx/127.1.170.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.170.129:41229
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194121577027-1706-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.170.129
--webserver_port=34051
--tserver_master_addrs=127.1.170.190:42111,127.1.170.189:40091,127.1.170.188:36059
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.170.129
--log_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194121577027-1706-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:09:40.821092 8651 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:09:40.822387 8651 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260923 20:09:40.834913 8717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:09:40.828067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.188:36059 (local address 127.0.0.1:41124)
0923 20:09:40.828705 (+ 638us) negotiation.cc:107] Waiting for socket to connect
0923 20:09:40.828729 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0923 20:09:40.828876 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:09:40.829956 (+ 1080us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:09:40.829964 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:09:40.829995 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:09:40.830143 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:09:40.830151 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:09:40.831396 (+ 1245us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:09:40.831404 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:09:40.832866 (+ 1462us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:09:40.832879 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:09:40.834078 (+ 1199us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:09:40.834085 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:09:40.834275 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:09:40.834312 (+ 37us) client_negotiation.cc:769] Sending connection context
0923 20:09:40.834420 (+ 108us) client_negotiation.cc:240] Negotiation successful
0923 20:09:40.834562 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":515,"thread_start_us":421,"threads_started":1}
I20260923 20:09:40.835016 8718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:09:40.828180 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41124 (local address 127.1.170.188:36059)
0923 20:09:40.829332 (+ 1152us) server_negotiation.cc:206] Beginning negotiation
0923 20:09:40.829338 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:09:40.829357 (+ 19us) server_negotiation.cc:407] Connection header received
0923 20:09:40.829428 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:09:40.829436 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:09:40.829535 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:09:40.829676 (+ 141us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:09:40.830283 (+ 607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:09:40.831231 (+ 948us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:09:40.833022 (+ 1791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:09:40.833800 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:09:40.834004 (+ 204us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:09:40.834218 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0923 20:09:40.834474 (+ 256us) server_negotiation.cc:299] Negotiation successful
0923 20:09:40.834621 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1014,"thread_start_us":895,"threads_started":1}
I20260923 20:09:40.835893 8674 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 0a16f1acbd7c4420b8d8cdf2732a5200 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260923 20:09:40.836831 8702 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0a16f1acbd7c4420b8d8cdf2732a5200 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f83afa4c3fa644f08196bf2ef04d733b. Latest consensus state: current_term: 1 leader_uuid: "f83afa4c3fa644f08196bf2ef04d733b" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f83afa4c3fa644f08196bf2ef04d733b" member_type: VOTER last_known_addr { host: "127.1.170.190" port: 42111 } } peers { permanent_uuid: "2fc4ff3aee844aa492e3b79e53ad59c3" member_type: VOTER last_known_addr { host: "127.1.170.189" port: 40091 } attrs { promote: false } } peers { permanent_uuid: "0a16f1acbd7c4420b8d8cdf2732a5200" member_type: NON_VOTER last_known_addr { host: "127.1.170.188" port: 36059 } attrs { promote: true } } }
I20260923 20:09:40.837153 8702 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0a16f1acbd7c4420b8d8cdf2732a5200 [sys.catalog]: This master's current role is: LEARNER
W20260923 20:09:40.838676 8720 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:09:40.838786 8721 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:09:40.838956 8723 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:09:40.840147 8651 server_base.cc:1034] running on GCE node
I20260923 20:09:40.840773 8651 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:09:40.841918 8651 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:09:40.843139 8651 hybrid_clock.cc:648] HybridClock initialized: now 1790194180843064 us; error 73 us; skew 500 ppm
I20260923 20:09:40.843473 8651 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:09:40 dist-test-slave-th4g krb5kdc[7830](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194180, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.170.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:09:40.846886 8651 init.cc:377] Logged in from keytab as oryx/127.1.170.129@KRBTEST.COM (short username oryx)
I20260923 20:09:40.848407 8651 webserver.cc:460] Webserver started at http://127.1.170.129:34051/ using document root <none> and password file <none>
I20260923 20:09:40.849164 8651 fs_manager.cc:359] Metadata directory not provided
I20260923 20:09:40.849325 8651 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:09:40.854746 8651 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260923 20:09:40.857467 8730 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:09:40.858939 8651 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 20:09:40.859113 8651 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194121577027-1706-0/minicluster-data/ts-0/data,/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194121577027-1706-0/minicluster-data/ts-0/wal
uuid: "e120ba40339e4897a3aa5180c4c45acf"
format_stamp: "Formatted at 2026-09-23 20:09:37 on dist-test-slave-th4g"
server_key: "f5b649f1f57f06efbb53de215791c76c"
server_key_iv: "45d7cb2b88063be3fee557c7c9d404f9"
server_key_version: "encryptionkey@0"
I20260923 20:09:40.860091 8651 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194121577027-1706-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194121577027-1706-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194121577027-1706-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:09:40.884357 8651 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:09:40.889573 8651 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:09:40.889935 8651 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:09:40.891079 8651 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 20:09:40.892590 8651 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 20:09:40.892727 8651 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 20:09:40.892863 8651 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 20:09:40.892967 8651 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 20:09:40.935091 8651 rpc_server.cc:307] RPC server started. Bound to: 127.1.170.129:41229
I20260923 20:09:40.935186 8842 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.170.129:41229 every 8 connection(s)
I20260923 20:09:40.936661 8651 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194121577027-1706-0/minicluster-data/ts-0/data/info.pb
I20260923 20:09:40.941084 1706 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqUuku6/build/asan/bin/kudu as pid 8651
I20260923 20:09:40.941426 1706 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskqUuku6/build/asan/bin/kudu pid 8070
Sep 23 20:09:40 dist-test-slave-th4g krb5kdc[7830](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194180, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.170.129@KRBTEST.COM for oryx/127.1.170.188@KRBTEST.COM
I20260923 20:09:40.960047 8718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:09:40.940777 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.170.129:46583 (local address 127.1.170.188:36059)
0923 20:09:40.941138 (+ 361us) server_negotiation.cc:206] Beginning negotiation
0923 20:09:40.941144 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:09:40.943251 (+ 2107us) server_negotiation.cc:407] Connection header received
0923 20:09:40.945809 (+ 2558us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:09:40.945814 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:09:40.945929 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:09:40.946049 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:09:40.947687 (+ 1638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:09:40.948533 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:09:40.950202 (+ 1669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:09:40.950615 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:09:40.950756 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:09:40.954416 (+ 3660us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:09:40.954422 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:09:40.954427 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:09:40.954469 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:09:40.956974 (+ 2505us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:09:40.957719 (+ 745us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:09:40.957724 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:09:40.957727 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:09:40.957849 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:09:40.958317 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:09:40.958322 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:09:40.958325 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:09:40.958594 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:09:40.958754 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0923 20:09:40.959457 (+ 703us) server_negotiation.cc:299] Negotiation successful
0923 20:09:40.959693 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":223}
I20260923 20:09:40.961284 8849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:09:40.940173 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.188:36059 (local address 127.1.170.129:46583)
0923 20:09:40.943071 (+ 2898us) negotiation.cc:107] Waiting for socket to connect
0923 20:09:40.943120 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0923 20:09:40.945538 (+ 2418us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:09:40.946295 (+ 757us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:09:40.946304 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:09:40.946807 (+ 503us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:09:40.947511 (+ 704us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:09:40.947528 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:09:40.948670 (+ 1142us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:09:40.948675 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:09:40.950039 (+ 1364us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:09:40.950060 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:09:40.950847 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:09:40.950853 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:09:40.951095 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:09:40.951804 (+ 709us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:09:40.951825 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:09:40.954212 (+ 2387us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:09:40.957120 (+ 2908us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:09:40.957130 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:09:40.957156 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:09:40.957587 (+ 431us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:09:40.957993 (+ 406us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:09:40.957998 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:09:40.958002 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:09:40.958188 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:09:40.958729 (+ 541us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:09:40.958737 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:09:40.959095 (+ 358us) client_negotiation.cc:769] Sending connection context
0923 20:09:40.959367 (+ 272us) client_negotiation.cc:240] Negotiation successful
0923 20:09:40.959906 (+ 539us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1695,"thread_start_us":463,"threads_started":1}
I20260923 20:09:40.962945 8844 heartbeater.cc:347] Connected to a master server at 127.1.170.188:36059
I20260923 20:09:40.963281 8844 heartbeater.cc:464] Registering TS with master...
I20260923 20:09:40.964236 8844 heartbeater.cc:511] Master 127.1.170.188:36059 requested a full tablet report, sending...
I20260923 20:09:40.966657 8664 ts_manager.cc:194] Registered new tserver with Master: e120ba40339e4897a3aa5180c4c45acf (127.1.170.129:41229)
Sep 23 20:09:40 dist-test-slave-th4g krb5kdc[7830](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194180, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.170.129@KRBTEST.COM for oryx/127.1.170.190@KRBTEST.COM
Sep 23 20:09:40 dist-test-slave-th4g krb5kdc[7830](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194176, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.1.170.130@KRBTEST.COM
Sep 23 20:09:40 dist-test-slave-th4g krb5kdc[7830](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194180, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.170.129@KRBTEST.COM for oryx/127.1.170.189@KRBTEST.COM
I20260923 20:09:40.985532 8848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:09:40.940432 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.170.129:39227 (local address 127.1.170.190:42111)
0923 20:09:40.940951 (+ 519us) server_negotiation.cc:206] Beginning negotiation
0923 20:09:40.940956 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 20:09:40.962972 (+ 22016us) server_negotiation.cc:407] Connection header received
0923 20:09:40.963051 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:09:40.963057 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:09:40.963138 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:09:40.963244 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:09:40.967968 (+ 4724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:09:40.969129 (+ 1161us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:09:40.971916 (+ 2787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:09:40.972539 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:09:40.972684 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:09:40.976620 (+ 3936us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:09:40.976626 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:09:40.976629 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:09:40.976694 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:09:40.980895 (+ 4201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:09:40.982390 (+ 1495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:09:40.982396 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:09:40.982401 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:09:40.982618 (+ 217us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:09:40.984017 (+ 1399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:09:40.984023 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:09:40.984026 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:09:40.984332 (+ 306us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:09:40.984514 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0923 20:09:40.984924 (+ 410us) server_negotiation.cc:299] Negotiation successful
0923 20:09:40.985147 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":422,"thread_start_us":337,"threads_started":1}
I20260923 20:09:40.985543 8849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:09:40.940171 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.190:42111 (local address 127.1.170.129:39227)
0923 20:09:40.962169 (+ 21998us) negotiation.cc:107] Waiting for socket to connect
0923 20:09:40.962233 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0923 20:09:40.962348 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:09:40.964985 (+ 2637us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:09:40.964991 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:09:40.965570 (+ 579us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:09:40.965787 (+ 217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:09:40.965801 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:09:40.969349 (+ 3548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:09:40.969355 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:09:40.971696 (+ 2341us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:09:40.971719 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:09:40.972704 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:09:40.972710 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:09:40.972911 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:09:40.973777 (+ 866us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:09:40.973801 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:09:40.976382 (+ 2581us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:09:40.981047 (+ 4665us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:09:40.981053 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:09:40.981058 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:09:40.982178 (+ 1120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:09:40.983712 (+ 1534us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:09:40.983716 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:09:40.983719 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:09:40.983890 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:09:40.984487 (+ 597us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:09:40.984492 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:09:40.984653 (+ 161us) client_negotiation.cc:769] Sending connection context
0923 20:09:40.984818 (+ 165us) client_negotiation.cc:240] Negotiation successful
0923 20:09:40.985135 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":21810,"mutex_wait_us":27,"spinlock_wait_cycles":71040,"thread_start_us":28142,"threads_started":1}
I20260923 20:09:40.986775 8843 heartbeater.cc:347] Connected to a master server at 127.1.170.190:42111
I20260923 20:09:40.986958 8843 heartbeater.cc:464] Registering TS with master...
I20260923 20:09:40.987818 8853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:09:40.942473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57908 (local address 127.1.170.130:36095)
0923 20:09:40.943083 (+ 610us) server_negotiation.cc:206] Beginning negotiation
0923 20:09:40.943092 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 20:09:40.965841 (+ 22749us) server_negotiation.cc:407] Connection header received
0923 20:09:40.965957 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:09:40.965969 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:09:40.966298 (+ 329us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:09:40.966492 (+ 194us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:09:40.968459 (+ 1967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:09:40.970184 (+ 1725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:09:40.971562 (+ 1378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:09:40.972314 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:09:40.972436 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:09:40.979046 (+ 6610us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:09:40.979068 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:09:40.979091 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:09:40.979146 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:09:40.983747 (+ 4601us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:09:40.984774 (+ 1027us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:09:40.984785 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:09:40.984791 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:09:40.984959 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:09:40.985693 (+ 734us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:09:40.985699 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:09:40.985702 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:09:40.986181 (+ 479us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:09:40.986493 (+ 312us) server_negotiation.cc:1035] Waiting for connection context
0923 20:09:40.987108 (+ 615us) server_negotiation.cc:299] Negotiation successful
0923 20:09:40.987373 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":472,"thread_start_us":321,"threads_started":1}
I20260923 20:09:40.987818 8843 heartbeater.cc:511] Master 127.1.170.190:42111 requested a full tablet report, sending...
I20260923 20:09:40.989137 7875 ts_manager.cc:194] Re-registered known tserver with Master: e120ba40339e4897a3aa5180c4c45acf (127.1.170.129:41229)
I20260923 20:09:40.989465 8194 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:09:40.990665 7875 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.1.170.129@KRBTEST.COM'} at 127.1.170.129:39227
I20260923 20:09:40.991006 8850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:09:40.941261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.189:40091 (local address 127.1.170.129:44309)
0923 20:09:40.970175 (+ 28914us) negotiation.cc:107] Waiting for socket to connect
0923 20:09:40.970196 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0923 20:09:40.970317 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:09:40.973081 (+ 2764us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:09:40.973087 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:09:40.973746 (+ 659us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:09:40.974098 (+ 352us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:09:40.974113 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:09:40.975769 (+ 1656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:09:40.975775 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:09:40.977204 (+ 1429us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:09:40.977215 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:09:40.978218 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:09:40.978226 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:09:40.978372 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:09:40.978936 (+ 564us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:09:40.978960 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:09:40.982554 (+ 3594us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:09:40.988186 (+ 5632us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:09:40.988191 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:09:40.988195 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:09:40.988707 (+ 512us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:09:40.989129 (+ 422us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:09:40.989134 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:09:40.989138 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:09:40.989283 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:09:40.990041 (+ 758us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:09:40.990072 (+ 31us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:09:40.990240 (+ 168us) client_negotiation.cc:769] Sending connection context
0923 20:09:40.990374 (+ 134us) client_negotiation.cc:240] Negotiation successful
0923 20:09:40.990655 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28416,"mutex_wait_us":328,"thread_start_us":30319,"threads_started":1}
I20260923 20:09:40.992547 8847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:09:40.939554 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.170.129:44309 (local address 127.1.170.189:40091)
0923 20:09:40.972522 (+ 32968us) server_negotiation.cc:206] Beginning negotiation
0923 20:09:40.972527 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 20:09:40.972543 (+ 16us) server_negotiation.cc:407] Connection header received
0923 20:09:40.972588 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:09:40.972592 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:09:40.972762 (+ 170us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:09:40.972876 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:09:40.974272 (+ 1396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:09:40.975565 (+ 1293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:09:40.977360 (+ 1795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:09:40.977807 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:09:40.977959 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:09:40.982924 (+ 4965us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:09:40.982930 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:09:40.982933 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:09:40.982972 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:09:40.986633 (+ 3661us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:09:40.988864 (+ 2231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:09:40.988870 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:09:40.988873 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:09:40.989022 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:09:40.989469 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:09:40.989474 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:09:40.989477 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:09:40.989866 (+ 389us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:09:40.991723 (+ 1857us) server_negotiation.cc:1035] Waiting for connection context
0923 20:09:40.991864 (+ 141us) server_negotiation.cc:299] Negotiation successful
0923 20:09:40.992119 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32860,"thread_start_us":32741,"threads_started":1}
I20260923 20:09:40.993479 8845 heartbeater.cc:347] Connected to a master server at 127.1.170.189:40091
I20260923 20:09:40.993639 8845 heartbeater.cc:464] Registering TS with master...
I20260923 20:09:40.994102 8845 heartbeater.cc:511] Master 127.1.170.189:40091 requested a full tablet report, sending...
I20260923 20:09:40.995739 8476 ts_manager.cc:194] Registered new tserver with Master: e120ba40339e4897a3aa5180c4c45acf (127.1.170.129:41229)
I20260923 20:09:41.077171 1706 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqUuku6/build/asan/bin/kudu with pid 8070
I20260923 20:09:41.094417 1706 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqUuku6/build/asan/bin/kudu
/tmp/dist-test-taskqUuku6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194121577027-1706-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194121577027-1706-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194121577027-1706-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194121577027-1706-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.170.130:36095
--local_ip_for_outbound_sockets=127.1.170.130
--tserver_master_addrs=127.1.170.190:42111
--webserver_port=42585
--webserver_interface=127.1.170.130
--keytab_file=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194121577027-1706-0/krb5kdc/oryx_127.1.170.130.keytab
--principal=oryx/127.1.170.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.170.148:44293
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.1.170.190:42111,127.1.170.189:40091
--tserver_master_addrs=127.1.170.190:42111,127.1.170.189:40091,127.1.170.188:36059 with env {KRB5CCNAME=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194121577027-1706-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194121577027-1706-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194121577027-1706-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260923 20:09:41.208458 8621 sys_catalog.cc:455] T 00000000000000000000000000000000 P f83afa4c3fa644f08196bf2ef04d733b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f83afa4c3fa644f08196bf2ef04d733b" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f83afa4c3fa644f08196bf2ef04d733b" member_type: VOTER last_known_addr { host: "127.1.170.190" port: 42111 } } peers { permanent_uuid: "2fc4ff3aee844aa492e3b79e53ad59c3" member_type: VOTER last_known_addr { host: "127.1.170.189" port: 40091 } attrs { promote: false } } peers { permanent_uuid: "0a16f1acbd7c4420b8d8cdf2732a5200" member_type: NON_VOTER last_known_addr { host: "127.1.170.188" port: 36059 } attrs { promote: true } } }
I20260923 20:09:41.208875 8621 sys_catalog.cc:458] T 00000000000000000000000000000000 P f83afa4c3fa644f08196bf2ef04d733b [sys.catalog]: This master's current role is: LEADER
I20260923 20:09:41.210175 8857 raft_consensus.cc:1050] T 00000000000000000000000000000000 P f83afa4c3fa644f08196bf2ef04d733b: attempting to promote NON_VOTER 0a16f1acbd7c4420b8d8cdf2732a5200 to VOTER
I20260923 20:09:41.210839 8857 consensus_queue.cc:237] T 00000000000000000000000000000000 P f83afa4c3fa644f08196bf2ef04d733b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "f83afa4c3fa644f08196bf2ef04d733b" member_type: VOTER last_known_addr { host: "127.1.170.190" port: 42111 } } peers { permanent_uuid: "2fc4ff3aee844aa492e3b79e53ad59c3" member_type: VOTER last_known_addr { host: "127.1.170.189" port: 40091 } attrs { promote: false } } peers { permanent_uuid: "0a16f1acbd7c4420b8d8cdf2732a5200" member_type: VOTER last_known_addr { host: "127.1.170.188" port: 36059 } attrs { promote: false } }
I20260923 20:09:41.212729 8674 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 0a16f1acbd7c4420b8d8cdf2732a5200 [term 1 LEARNER]: Refusing update from remote peer f83afa4c3fa644f08196bf2ef04d733b: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260923 20:09:41.212724 8485 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 2fc4ff3aee844aa492e3b79e53ad59c3 [term 1 FOLLOWER]: Refusing update from remote peer f83afa4c3fa644f08196bf2ef04d733b: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260923 20:09:41.213366 8858 consensus_queue.cc:1035] T 00000000000000000000000000000000 P f83afa4c3fa644f08196bf2ef04d733b [LEADER]: Connected to new peer: Peer: permanent_uuid: "2fc4ff3aee844aa492e3b79e53ad59c3" member_type: VOTER last_known_addr { host: "127.1.170.189" port: 40091 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260923 20:09:41.213810 8857 consensus_queue.cc:1035] T 00000000000000000000000000000000 P f83afa4c3fa644f08196bf2ef04d733b [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a16f1acbd7c4420b8d8cdf2732a5200" member_type: VOTER last_known_addr { host: "127.1.170.188" port: 36059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260923 20:09:41.217039 8857 raft_consensus.cc:2932] T 00000000000000000000000000000000 P f83afa4c3fa644f08196bf2ef04d733b [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 0a16f1acbd7c4420b8d8cdf2732a5200 (127.1.170.188) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "f83afa4c3fa644f08196bf2ef04d733b" member_type: VOTER last_known_addr { host: "127.1.170.190" port: 42111 } } peers { permanent_uuid: "2fc4ff3aee844aa492e3b79e53ad59c3" member_type: VOTER last_known_addr { host: "127.1.170.189" port: 40091 } attrs { promote: false } } peers { permanent_uuid: "0a16f1acbd7c4420b8d8cdf2732a5200" member_type: VOTER last_known_addr { host: "127.1.170.188" port: 36059 } attrs { promote: false } } }
I20260923 20:09:41.217690 8485 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 2fc4ff3aee844aa492e3b79e53ad59c3 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 0a16f1acbd7c4420b8d8cdf2732a5200 (127.1.170.188) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "f83afa4c3fa644f08196bf2ef04d733b" member_type: VOTER last_known_addr { host: "127.1.170.190" port: 42111 } } peers { permanent_uuid: "2fc4ff3aee844aa492e3b79e53ad59c3" member_type: VOTER last_known_addr { host: "127.1.170.189" port: 40091 } attrs { promote: false } } peers { permanent_uuid: "0a16f1acbd7c4420b8d8cdf2732a5200" member_type: VOTER last_known_addr { host: "127.1.170.188" port: 36059 } attrs { promote: false } } }
I20260923 20:09:41.220741 8861 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2fc4ff3aee844aa492e3b79e53ad59c3 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f83afa4c3fa644f08196bf2ef04d733b" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "f83afa4c3fa644f08196bf2ef04d733b" member_type: VOTER last_known_addr { host: "127.1.170.190" port: 42111 } } peers { permanent_uuid: "2fc4ff3aee844aa492e3b79e53ad59c3" member_type: VOTER last_known_addr { host: "127.1.170.189" port: 40091 } attrs { promote: false } } peers { permanent_uuid: "0a16f1acbd7c4420b8d8cdf2732a5200" member_type: VOTER last_known_addr { host: "127.1.170.188" port: 36059 } attrs { promote: false } } }
I20260923 20:09:41.221273 8861 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2fc4ff3aee844aa492e3b79e53ad59c3 [sys.catalog]: This master's current role is: FOLLOWER
I20260923 20:09:41.222762 8674 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 0a16f1acbd7c4420b8d8cdf2732a5200 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 0a16f1acbd7c4420b8d8cdf2732a5200 (127.1.170.188) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "f83afa4c3fa644f08196bf2ef04d733b" member_type: VOTER last_known_addr { host: "127.1.170.190" port: 42111 } } peers { permanent_uuid: "2fc4ff3aee844aa492e3b79e53ad59c3" member_type: VOTER last_known_addr { host: "127.1.170.189" port: 40091 } attrs { promote: false } } peers { permanent_uuid: "0a16f1acbd7c4420b8d8cdf2732a5200" member_type: VOTER last_known_addr { host: "127.1.170.188" port: 36059 } attrs { promote: false } } }
I20260923 20:09:41.228893 8702 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0a16f1acbd7c4420b8d8cdf2732a5200 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f83afa4c3fa644f08196bf2ef04d733b" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "f83afa4c3fa644f08196bf2ef04d733b" member_type: VOTER last_known_addr { host: "127.1.170.190" port: 42111 } } peers { permanent_uuid: "2fc4ff3aee844aa492e3b79e53ad59c3" member_type: VOTER last_known_addr { host: "127.1.170.189" port: 40091 } attrs { promote: false } } peers { permanent_uuid: "0a16f1acbd7c4420b8d8cdf2732a5200" member_type: VOTER last_known_addr { host: "127.1.170.188" port: 36059 } attrs { promote: false } } }
I20260923 20:09:41.229278 8702 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0a16f1acbd7c4420b8d8cdf2732a5200 [sys.catalog]: This master's current role is: FOLLOWER
I20260923 20:09:41.233232 8621 sys_catalog.cc:455] T 00000000000000000000000000000000 P f83afa4c3fa644f08196bf2ef04d733b [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "f83afa4c3fa644f08196bf2ef04d733b" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "f83afa4c3fa644f08196bf2ef04d733b" member_type: VOTER last_known_addr { host: "127.1.170.190" port: 42111 } } peers { permanent_uuid: "2fc4ff3aee844aa492e3b79e53ad59c3" member_type: VOTER last_known_addr { host: "127.1.170.189" port: 40091 } attrs { promote: false } } peers { permanent_uuid: "0a16f1acbd7c4420b8d8cdf2732a5200" member_type: VOTER last_known_addr { host: "127.1.170.188" port: 36059 } attrs { promote: false } } }
I20260923 20:09:41.233532 8621 sys_catalog.cc:458] T 00000000000000000000000000000000 P f83afa4c3fa644f08196bf2ef04d733b [sys.catalog]: This master's current role is: LEADER
I20260923 20:09:41.233846 8857 sys_catalog.cc:455] T 00000000000000000000000000000000 P f83afa4c3fa644f08196bf2ef04d733b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f83afa4c3fa644f08196bf2ef04d733b" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "f83afa4c3fa644f08196bf2ef04d733b" member_type: VOTER last_known_addr { host: "127.1.170.190" port: 42111 } } peers { permanent_uuid: "2fc4ff3aee844aa492e3b79e53ad59c3" member_type: VOTER last_known_addr { host: "127.1.170.189" port: 40091 } attrs { promote: false } } peers { permanent_uuid: "0a16f1acbd7c4420b8d8cdf2732a5200" member_type: VOTER last_known_addr { host: "127.1.170.188" port: 36059 } attrs { promote: false } } }
I20260923 20:09:41.234108 8857 sys_catalog.cc:458] T 00000000000000000000000000000000 P f83afa4c3fa644f08196bf2ef04d733b [sys.catalog]: This master's current role is: LEADER
I20260923 20:09:41.234303 8621 sys_catalog.cc:455] T 00000000000000000000000000000000 P f83afa4c3fa644f08196bf2ef04d733b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f83afa4c3fa644f08196bf2ef04d733b" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "f83afa4c3fa644f08196bf2ef04d733b" member_type: VOTER last_known_addr { host: "127.1.170.190" port: 42111 } } peers { permanent_uuid: "2fc4ff3aee844aa492e3b79e53ad59c3" member_type: VOTER last_known_addr { host: "127.1.170.189" port: 40091 } attrs { promote: false } } peers { permanent_uuid: "0a16f1acbd7c4420b8d8cdf2732a5200" member_type: VOTER last_known_addr { host: "127.1.170.188" port: 36059 } attrs { promote: false } } }
I20260923 20:09:41.234539 8621 sys_catalog.cc:458] T 00000000000000000000000000000000 P f83afa4c3fa644f08196bf2ef04d733b [sys.catalog]: This master's current role is: LEADER
W20260923 20:09:41.448199 8856 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:09:41.448495 8856 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:09:41.448647 8856 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:09:41.456717 8856 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:09:41.456853 8856 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:09:41.456971 8856 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.170.130
I20260923 20:09:41.468809 8856 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.170.148:44293
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194121577027-1706-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194121577027-1706-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194121577027-1706-0/krb5kdc/oryx_127.1.170.130.keytab
--principal=oryx/127.1.170.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.170.130:36095
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194121577027-1706-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.170.130
--webserver_port=42585
--tserver_master_addrs=127.1.170.190:42111,127.1.170.189:40091,127.1.170.188:36059
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.170.130
--log_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194121577027-1706-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:09:41.470432 8856 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:09:41.472077 8856 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:09:41.488691 8868 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:09:41.488759 8869 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:09:41.488823 8871 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:09:41.489564 8856 server_base.cc:1034] running on GCE node
I20260923 20:09:41.490360 8856 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:09:41.491459 8856 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:09:41.492735 8856 hybrid_clock.cc:648] HybridClock initialized: now 1790194181492679 us; error 69 us; skew 500 ppm
I20260923 20:09:41.493117 8856 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:09:41 dist-test-slave-th4g krb5kdc[7830](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194181, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.170.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:09:41.497413 8856 init.cc:377] Logged in from keytab as oryx/127.1.170.130@KRBTEST.COM (short username oryx)
I20260923 20:09:41.499624 8856 webserver.cc:460] Webserver started at http://127.1.170.130:42585/ using document root <none> and password file <none>
I20260923 20:09:41.500458 8856 fs_manager.cc:359] Metadata directory not provided
I20260923 20:09:41.500589 8856 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:09:41.508028 8856 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.001s sys 0.003s
I20260923 20:09:41.511997 8878 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:09:41.513681 8856 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.000s sys 0.001s
I20260923 20:09:41.513899 8856 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194121577027-1706-0/minicluster-data/ts-1/data,/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194121577027-1706-0/minicluster-data/ts-1/wal
uuid: "fee8671d00d44381ac37762a73d5ae51"
format_stamp: "Formatted at 2026-09-23 20:09:38 on dist-test-slave-th4g"
server_key: "a7d86ea9c6f6910ab02cb987366e5ded"
server_key_iv: "27ba946d1753dfd2ec27652629cad58e"
server_key_version: "encryptionkey@0"
I20260923 20:09:41.515017 8856 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194121577027-1706-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194121577027-1706-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194121577027-1706-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:09:41.546653 8856 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:09:41.551288 8856 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:09:41.551596 8856 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:09:41.552775 8856 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 20:09:41.554332 8856 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 20:09:41.554420 8856 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 20:09:41.554569 8856 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 20:09:41.554649 8856 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 20:09:41.590940 8856 rpc_server.cc:307] RPC server started. Bound to: 127.1.170.130:36095
I20260923 20:09:41.591066 8990 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.170.130:36095 every 8 connection(s)
I20260923 20:09:41.592407 8856 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194121577027-1706-0/minicluster-data/ts-1/data/info.pb
I20260923 20:09:41.599108 1706 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqUuku6/build/asan/bin/kudu as pid 8856
I20260923 20:09:41.599901 1706 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskqUuku6/build/asan/bin/kudu pid 8204
Sep 23 20:09:41 dist-test-slave-th4g krb5kdc[7830](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194181, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.170.130@KRBTEST.COM for oryx/127.1.170.188@KRBTEST.COM
Sep 23 20:09:41 dist-test-slave-th4g krb5kdc[7830](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194176, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.1.170.131@KRBTEST.COM
Sep 23 20:09:41 dist-test-slave-th4g krb5kdc[7830](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194181, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.170.130@KRBTEST.COM for oryx/127.1.170.189@KRBTEST.COM
Sep 23 20:09:41 dist-test-slave-th4g krb5kdc[7830](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194181, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.170.130@KRBTEST.COM for oryx/127.1.170.190@KRBTEST.COM
I20260923 20:09:41.633441 8996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:09:41.595777 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.170.130:47621 (local address 127.1.170.189:40091)
0923 20:09:41.596770 (+ 993us) server_negotiation.cc:206] Beginning negotiation
0923 20:09:41.596776 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:09:41.608550 (+ 11774us) server_negotiation.cc:407] Connection header received
0923 20:09:41.608995 (+ 445us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:09:41.609001 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:09:41.609114 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:09:41.609208 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:09:41.612581 (+ 3373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:09:41.613665 (+ 1084us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:09:41.617027 (+ 3362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:09:41.617729 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:09:41.617847 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:09:41.624749 (+ 6902us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:09:41.624756 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:09:41.624761 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:09:41.624804 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:09:41.629145 (+ 4341us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:09:41.630499 (+ 1354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:09:41.630505 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:09:41.630508 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:09:41.630654 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:09:41.631213 (+ 559us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:09:41.631218 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:09:41.631221 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:09:41.631594 (+ 373us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:09:41.631728 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0923 20:09:41.632821 (+ 1093us) server_negotiation.cc:299] Negotiation successful
0923 20:09:41.633049 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":890,"thread_start_us":778,"threads_started":1}
I20260923 20:09:41.633836 8997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:09:41.596079 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.170.130:57061 (local address 127.1.170.190:42111)
0923 20:09:41.596571 (+ 492us) server_negotiation.cc:206] Beginning negotiation
0923 20:09:41.596578 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:09:41.608801 (+ 12223us) server_negotiation.cc:407] Connection header received
0923 20:09:41.609020 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:09:41.609026 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:09:41.609101 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:09:41.609191 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:09:41.612089 (+ 2898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:09:41.613370 (+ 1281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:09:41.617131 (+ 3761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:09:41.617837 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:09:41.618001 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:09:41.625618 (+ 7617us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:09:41.625628 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:09:41.625637 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:09:41.625683 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:09:41.629783 (+ 4100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:09:41.630711 (+ 928us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:09:41.630719 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:09:41.630723 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:09:41.630871 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:09:41.631329 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:09:41.631335 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:09:41.631340 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:09:41.631697 (+ 357us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:09:41.631828 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0923 20:09:41.633220 (+ 1392us) server_negotiation.cc:299] Negotiation successful
0923 20:09:41.633458 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":359,"thread_start_us":255,"threads_started":1}
I20260923 20:09:41.634676 9001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:09:41.598397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.170.130:42005 (local address 127.1.170.188:36059)
0923 20:09:41.610341 (+ 11944us) server_negotiation.cc:206] Beginning negotiation
0923 20:09:41.610346 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 20:09:41.610369 (+ 23us) server_negotiation.cc:407] Connection header received
0923 20:09:41.610427 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:09:41.610434 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:09:41.610514 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:09:41.610845 (+ 331us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:09:41.612068 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:09:41.613267 (+ 1199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:09:41.617041 (+ 3774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:09:41.617746 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:09:41.618465 (+ 719us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:09:41.625138 (+ 6673us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:09:41.625146 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:09:41.625151 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:09:41.625193 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:09:41.629145 (+ 3952us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:09:41.630499 (+ 1354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:09:41.630505 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:09:41.630508 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:09:41.630655 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:09:41.631214 (+ 559us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:09:41.631218 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:09:41.631222 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:09:41.631578 (+ 356us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:09:41.631720 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0923 20:09:41.634066 (+ 2346us) server_negotiation.cc:299] Negotiation successful
0923 20:09:41.634292 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11801,"thread_start_us":11654,"threads_started":1}
I20260923 20:09:41.634781 9003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:09:41.601375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52358 (local address 127.1.170.131:34805)
0923 20:09:41.602028 (+ 653us) server_negotiation.cc:206] Beginning negotiation
0923 20:09:41.602040 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0923 20:09:41.606537 (+ 4497us) server_negotiation.cc:407] Connection header received
0923 20:09:41.606921 (+ 384us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:09:41.606938 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:09:41.607304 (+ 366us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:09:41.607478 (+ 174us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:09:41.610405 (+ 2927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:09:41.612032 (+ 1627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:09:41.614166 (+ 2134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:09:41.615214 (+ 1048us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:09:41.615369 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:09:41.624332 (+ 8963us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:09:41.624355 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:09:41.624375 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:09:41.624425 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:09:41.629624 (+ 5199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:09:41.630609 (+ 985us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:09:41.630621 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:09:41.630624 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:09:41.630790 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:09:41.631295 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:09:41.631302 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:09:41.631308 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:09:41.632104 (+ 796us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:09:41.632405 (+ 301us) server_negotiation.cc:1035] Waiting for connection context
0923 20:09:41.634073 (+ 1668us) server_negotiation.cc:299] Negotiation successful
0923 20:09:41.634384 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":436,"thread_start_us":307,"threads_started":1}
I20260923 20:09:41.635386 8998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:09:41.596526 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.188:36059 (local address 127.1.170.130:42005)
0923 20:09:41.597788 (+ 1262us) negotiation.cc:107] Waiting for socket to connect
0923 20:09:41.597844 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0923 20:09:41.600925 (+ 3081us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:09:41.610715 (+ 9790us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:09:41.610719 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:09:41.611182 (+ 463us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:09:41.611866 (+ 684us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:09:41.611898 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:09:41.613446 (+ 1548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:09:41.613451 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:09:41.616875 (+ 3424us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:09:41.616884 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:09:41.618054 (+ 1170us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:09:41.618064 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:09:41.618488 (+ 424us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:09:41.619550 (+ 1062us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:09:41.619580 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:09:41.624846 (+ 5266us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:09:41.629345 (+ 4499us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:09:41.629356 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:09:41.629377 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:09:41.630348 (+ 971us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:09:41.630798 (+ 450us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:09:41.630803 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:09:41.630807 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:09:41.631090 (+ 283us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:09:41.631809 (+ 719us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:09:41.631817 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:09:41.632356 (+ 539us) client_negotiation.cc:769] Sending connection context
0923 20:09:41.632697 (+ 341us) client_negotiation.cc:240] Negotiation successful
0923 20:09:41.633306 (+ 609us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":942,"mutex_wait_us":589,"spinlock_wait_cycles":39168,"thread_start_us":684,"threads_started":1}
I20260923 20:09:41.635413 8999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:09:41.597003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.189:40091 (local address 127.1.170.130:47621)
0923 20:09:41.608402 (+ 11399us) negotiation.cc:107] Waiting for socket to connect
0923 20:09:41.608427 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0923 20:09:41.608832 (+ 405us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:09:41.609715 (+ 883us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:09:41.609729 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:09:41.612120 (+ 2391us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:09:41.612390 (+ 270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:09:41.612402 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:09:41.613862 (+ 1460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:09:41.613869 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:09:41.616836 (+ 2967us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:09:41.616854 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:09:41.618199 (+ 1345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:09:41.618205 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:09:41.618940 (+ 735us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:09:41.619702 (+ 762us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:09:41.619724 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:09:41.624445 (+ 4721us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:09:41.629345 (+ 4900us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:09:41.629356 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:09:41.629377 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:09:41.630348 (+ 971us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:09:41.630798 (+ 450us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:09:41.630803 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:09:41.630807 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:09:41.631074 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:09:41.631809 (+ 735us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:09:41.631817 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:09:41.632356 (+ 539us) client_negotiation.cc:769] Sending connection context
0923 20:09:41.633931 (+ 1575us) client_negotiation.cc:240] Negotiation successful
0923 20:09:41.634204 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11232,"mutex_wait_us":237,"spinlock_wait_cycles":54144,"thread_start_us":11012,"threads_started":1}
I20260923 20:09:41.636238 8328 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:09:41.637188 8994 heartbeater.cc:347] Connected to a master server at 127.1.170.189:40091
I20260923 20:09:41.637542 8994 heartbeater.cc:464] Registering TS with master...
I20260923 20:09:41.637966 8993 heartbeater.cc:347] Connected to a master server at 127.1.170.188:36059
I20260923 20:09:41.638095 8993 heartbeater.cc:464] Registering TS with master...
I20260923 20:09:41.638964 8994 heartbeater.cc:511] Master 127.1.170.189:40091 requested a full tablet report, sending...
I20260923 20:09:41.639003 8993 heartbeater.cc:511] Master 127.1.170.188:36059 requested a full tablet report, sending...
I20260923 20:09:41.640316 9000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:09:41.597000 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.190:42111 (local address 127.1.170.130:57061)
0923 20:09:41.608690 (+ 11690us) negotiation.cc:107] Waiting for socket to connect
0923 20:09:41.608712 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0923 20:09:41.608891 (+ 179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:09:41.609715 (+ 824us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:09:41.609729 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:09:41.611049 (+ 1320us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:09:41.611870 (+ 821us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:09:41.611899 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:09:41.613569 (+ 1670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:09:41.613574 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:09:41.616927 (+ 3353us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:09:41.616948 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:09:41.618380 (+ 1432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:09:41.618388 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:09:41.618602 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:09:41.619340 (+ 738us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:09:41.619377 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:09:41.625326 (+ 5949us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:09:41.629986 (+ 4660us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:09:41.629992 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:09:41.629995 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:09:41.630569 (+ 574us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:09:41.631014 (+ 445us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:09:41.631022 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:09:41.631028 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:09:41.631209 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:09:41.631909 (+ 700us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:09:41.631913 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:09:41.632359 (+ 446us) client_negotiation.cc:769] Sending connection context
0923 20:09:41.634343 (+ 1984us) client_negotiation.cc:240] Negotiation successful
0923 20:09:41.634636 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11560,"mutex_wait_us":310,"spinlock_wait_cycles":52992,"thread_start_us":11488,"threads_started":1}
I20260923 20:09:41.641122 8476 ts_manager.cc:194] Registered new tserver with Master: fee8671d00d44381ac37762a73d5ae51 (127.1.170.130:36095)
I20260923 20:09:41.641151 8991 heartbeater.cc:347] Connected to a master server at 127.1.170.190:42111
I20260923 20:09:41.641268 8991 heartbeater.cc:464] Registering TS with master...
I20260923 20:09:41.641331 8664 ts_manager.cc:194] Registered new tserver with Master: fee8671d00d44381ac37762a73d5ae51 (127.1.170.130:36095)
I20260923 20:09:41.641789 8991 heartbeater.cc:511] Master 127.1.170.190:42111 requested a full tablet report, sending...
I20260923 20:09:41.642803 7875 ts_manager.cc:194] Re-registered known tserver with Master: fee8671d00d44381ac37762a73d5ae51 (127.1.170.130:36095)
I20260923 20:09:41.643663 7875 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.1.170.130@KRBTEST.COM'} at 127.1.170.130:57061
I20260923 20:09:41.728955 1706 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqUuku6/build/asan/bin/kudu with pid 8204
I20260923 20:09:41.746327 1706 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqUuku6/build/asan/bin/kudu
/tmp/dist-test-taskqUuku6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194121577027-1706-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194121577027-1706-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194121577027-1706-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194121577027-1706-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.170.131:34805
--local_ip_for_outbound_sockets=127.1.170.131
--tserver_master_addrs=127.1.170.190:42111
--webserver_port=40925
--webserver_interface=127.1.170.131
--keytab_file=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194121577027-1706-0/krb5kdc/oryx_127.1.170.131.keytab
--principal=oryx/127.1.170.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.170.148:44293
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.1.170.190:42111,127.1.170.189:40091
--tserver_master_addrs=127.1.170.190:42111,127.1.170.189:40091,127.1.170.188:36059 with env {KRB5CCNAME=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194121577027-1706-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194121577027-1706-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194121577027-1706-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260923 20:09:41.994093 8843 heartbeater.cc:503] Master 127.1.170.190:42111 was elected leader, sending a full tablet report...
W20260923 20:09:42.093294 9005 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:09:42.093602 9005 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:09:42.093696 9005 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:09:42.101727 9005 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:09:42.101852 9005 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:09:42.102032 9005 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.170.131
I20260923 20:09:42.112321 9005 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.170.148:44293
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194121577027-1706-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194121577027-1706-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194121577027-1706-0/krb5kdc/oryx_127.1.170.131.keytab
--principal=oryx/127.1.170.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.170.131:34805
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194121577027-1706-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.170.131
--webserver_port=40925
--tserver_master_addrs=127.1.170.190:42111,127.1.170.189:40091,127.1.170.188:36059
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.170.131
--log_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194121577027-1706-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:09:42.113848 9005 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:09:42.115218 9005 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:09:42.128983 9014 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:09:42.128950 9012 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:09:42.129364 9011 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:09:42.130323 9005 server_base.cc:1034] running on GCE node
I20260923 20:09:42.130970 9005 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:09:42.132012 9005 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:09:42.133296 9005 hybrid_clock.cc:648] HybridClock initialized: now 1790194182133237 us; error 69 us; skew 500 ppm
I20260923 20:09:42.133678 9005 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:09:42 dist-test-slave-th4g krb5kdc[7830](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194182, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.170.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:09:42.142329 9005 init.cc:377] Logged in from keytab as oryx/127.1.170.131@KRBTEST.COM (short username oryx)
I20260923 20:09:42.144109 9005 webserver.cc:460] Webserver started at http://127.1.170.131:40925/ using document root <none> and password file <none>
I20260923 20:09:42.144925 9005 fs_manager.cc:359] Metadata directory not provided
I20260923 20:09:42.145089 9005 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:09:42.150992 9005 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260923 20:09:42.154188 9021 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:09:42.155615 9005 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260923 20:09:42.155817 9005 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194121577027-1706-0/minicluster-data/ts-2/data,/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194121577027-1706-0/minicluster-data/ts-2/wal
uuid: "0ecb00af331c4c6eb682ec0f64fa0e47"
format_stamp: "Formatted at 2026-09-23 20:09:38 on dist-test-slave-th4g"
server_key: "a5da5edc9efa612cdad9dae6607d4e32"
server_key_iv: "e01d02b3ac3d282fe6d389699ecd0263"
server_key_version: "encryptionkey@0"
I20260923 20:09:42.156879 9005 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194121577027-1706-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194121577027-1706-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194121577027-1706-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:09:42.183013 9005 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:09:42.187618 9005 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:09:42.187950 9005 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:09:42.189409 9005 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 20:09:42.191524 9005 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 20:09:42.191653 9005 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 20:09:42.191802 9005 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 20:09:42.191907 9005 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 20:09:42.231107 9005 rpc_server.cc:307] RPC server started. Bound to: 127.1.170.131:34805
I20260923 20:09:42.231213 9134 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.170.131:34805 every 8 connection(s)
I20260923 20:09:42.232640 9005 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194121577027-1706-0/minicluster-data/ts-2/data/info.pb
I20260923 20:09:42.243309 1706 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqUuku6/build/asan/bin/kudu as pid 9005
Sep 23 20:09:42 dist-test-slave-th4g krb5kdc[7830](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194182, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.170.131@KRBTEST.COM for oryx/127.1.170.188@KRBTEST.COM
Sep 23 20:09:42 dist-test-slave-th4g krb5kdc[7830](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194182, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.170.131@KRBTEST.COM for oryx/127.1.170.190@KRBTEST.COM
I20260923 20:09:42.271296 9140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:09:42.236297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.170.131:44891 (local address 127.1.170.188:36059)
0923 20:09:42.247636 (+ 11339us) server_negotiation.cc:206] Beginning negotiation
0923 20:09:42.247641 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 20:09:42.247660 (+ 19us) server_negotiation.cc:407] Connection header received
0923 20:09:42.247711 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:09:42.247716 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:09:42.247801 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:09:42.247950 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:09:42.250623 (+ 2673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:09:42.251868 (+ 1245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:09:42.254798 (+ 2930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:09:42.255547 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:09:42.255669 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:09:42.262052 (+ 6383us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:09:42.262063 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:09:42.262070 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:09:42.262125 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:09:42.266675 (+ 4550us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:09:42.267943 (+ 1268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:09:42.267951 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:09:42.267958 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:09:42.268119 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:09:42.268759 (+ 640us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:09:42.268767 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:09:42.268773 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:09:42.269182 (+ 409us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:09:42.269323 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0923 20:09:42.270469 (+ 1146us) server_negotiation.cc:299] Negotiation successful
0923 20:09:42.270772 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11229,"thread_start_us":11123,"threads_started":1}
I20260923 20:09:42.273180 9141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:09:42.236297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.170.131:47425 (local address 127.1.170.190:42111)
0923 20:09:42.247219 (+ 10922us) server_negotiation.cc:206] Beginning negotiation
0923 20:09:42.247226 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:09:42.248231 (+ 1005us) server_negotiation.cc:407] Connection header received
0923 20:09:42.248552 (+ 321us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:09:42.248560 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:09:42.248678 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:09:42.248795 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:09:42.251633 (+ 2838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:09:42.252659 (+ 1026us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:09:42.255738 (+ 3079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:09:42.256611 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:09:42.256733 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:09:42.262160 (+ 5427us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:09:42.262170 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:09:42.262178 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:09:42.262232 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:09:42.266919 (+ 4687us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:09:42.268789 (+ 1870us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:09:42.268799 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:09:42.268806 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:09:42.269024 (+ 218us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:09:42.269966 (+ 942us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:09:42.269974 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:09:42.269980 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:09:42.270487 (+ 507us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:09:42.270659 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0923 20:09:42.272178 (+ 1519us) server_negotiation.cc:299] Negotiation successful
0923 20:09:42.272507 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10790,"thread_start_us":10929,"threads_started":1}
I20260923 20:09:42.274518 9142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:09:42.236769 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.188:36059 (local address 127.1.170.131:44891)
0923 20:09:42.242824 (+ 6055us) negotiation.cc:107] Waiting for socket to connect
0923 20:09:42.242891 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0923 20:09:42.246596 (+ 3705us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:09:42.248281 (+ 1685us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:09:42.248294 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:09:42.249377 (+ 1083us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:09:42.250326 (+ 949us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:09:42.250380 (+ 54us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:09:42.252070 (+ 1690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:09:42.252075 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:09:42.254595 (+ 2520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:09:42.254615 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:09:42.256092 (+ 1477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:09:42.256102 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:09:42.256482 (+ 380us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:09:42.257412 (+ 930us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:09:42.257450 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:09:42.261174 (+ 3724us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:09:42.266898 (+ 5724us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:09:42.266913 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:09:42.266939 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:09:42.267769 (+ 830us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:09:42.268275 (+ 506us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:09:42.268282 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:09:42.268289 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:09:42.268605 (+ 316us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:09:42.269419 (+ 814us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:09:42.269427 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:09:42.269970 (+ 543us) client_negotiation.cc:769] Sending connection context
0923 20:09:42.272015 (+ 2045us) client_negotiation.cc:240] Negotiation successful
0923 20:09:42.272447 (+ 432us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5652,"mutex_wait_us":4,"spinlock_wait_cycles":2432,"thread_start_us":452,"threads_started":1}
I20260923 20:09:42.276762 9136 heartbeater.cc:347] Connected to a master server at 127.1.170.188:36059
I20260923 20:09:42.276934 9143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:09:42.236769 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.190:42111 (local address 127.1.170.131:47425)
0923 20:09:42.248066 (+ 11297us) negotiation.cc:107] Waiting for socket to connect
0923 20:09:42.248092 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0923 20:09:42.248369 (+ 277us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:09:42.249033 (+ 664us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:09:42.249041 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:09:42.250388 (+ 1347us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:09:42.250736 (+ 348us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:09:42.250749 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:09:42.252852 (+ 2103us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:09:42.252860 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:09:42.255540 (+ 2680us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:09:42.255559 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:09:42.257494 (+ 1935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:09:42.257503 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:09:42.257772 (+ 269us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:09:42.258556 (+ 784us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:09:42.258587 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:09:42.261722 (+ 3135us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:09:42.267149 (+ 5427us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:09:42.267157 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:09:42.267164 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:09:42.268571 (+ 1407us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:09:42.269548 (+ 977us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:09:42.269555 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:09:42.269561 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:09:42.269787 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:09:42.271143 (+ 1356us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:09:42.271151 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:09:42.271365 (+ 214us) client_negotiation.cc:769] Sending connection context
0923 20:09:42.271563 (+ 198us) client_negotiation.cc:240] Negotiation successful
0923 20:09:42.276291 (+ 4728us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11070,"mutex_wait_us":2277,"spinlock_wait_cycles":2304,"thread_start_us":505,"threads_started":1}
I20260923 20:09:42.277117 9136 heartbeater.cc:464] Registering TS with master...
I20260923 20:09:42.278139 9136 heartbeater.cc:511] Master 127.1.170.188:36059 requested a full tablet report, sending...
I20260923 20:09:42.278327 9135 heartbeater.cc:347] Connected to a master server at 127.1.170.190:42111
I20260923 20:09:42.278479 9135 heartbeater.cc:464] Registering TS with master...
I20260923 20:09:42.279004 9135 heartbeater.cc:511] Master 127.1.170.190:42111 requested a full tablet report, sending...
I20260923 20:09:42.280227 7875 ts_manager.cc:194] Re-registered known tserver with Master: 0ecb00af331c4c6eb682ec0f64fa0e47 (127.1.170.131:34805)
I20260923 20:09:42.280406 8664 ts_manager.cc:194] Registered new tserver with Master: 0ecb00af331c4c6eb682ec0f64fa0e47 (127.1.170.131:34805)
I20260923 20:09:42.281339 7875 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.1.170.131@KRBTEST.COM'} at 127.1.170.131:47425
Sep 23 20:09:42 dist-test-slave-th4g krb5kdc[7830](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194182, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.170.131@KRBTEST.COM for oryx/127.1.170.189@KRBTEST.COM
I20260923 20:09:42.296944 9144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:09:42.236971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.189:40091 (local address 127.1.170.131:46089)
0923 20:09:42.247225 (+ 10254us) negotiation.cc:107] Waiting for socket to connect
0923 20:09:42.247257 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0923 20:09:42.247458 (+ 201us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:09:42.248281 (+ 823us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:09:42.248294 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:09:42.250937 (+ 2643us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:09:42.251407 (+ 470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:09:42.251424 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:09:42.269177 (+ 17753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:09:42.269183 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:09:42.273187 (+ 4004us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:09:42.273212 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:09:42.274540 (+ 1328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:09:42.274548 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:09:42.274741 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:09:42.282115 (+ 7374us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:09:42.282145 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:09:42.287680 (+ 5535us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:09:42.293736 (+ 6056us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:09:42.293742 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:09:42.293749 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:09:42.294365 (+ 616us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:09:42.294874 (+ 509us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:09:42.294881 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:09:42.294886 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:09:42.295036 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:09:42.295717 (+ 681us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:09:42.295725 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:09:42.295922 (+ 197us) client_negotiation.cc:769] Sending connection context
0923 20:09:42.296058 (+ 136us) client_negotiation.cc:240] Negotiation successful
0923 20:09:42.296435 (+ 377us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10146,"spinlock_wait_cycles":68864,"thread_start_us":10714,"threads_started":1}
I20260923 20:09:42.298195 9139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:09:42.235989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.170.131:46089 (local address 127.1.170.189:40091)
0923 20:09:42.236503 (+ 514us) server_negotiation.cc:206] Beginning negotiation
0923 20:09:42.236513 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0923 20:09:42.247339 (+ 10826us) server_negotiation.cc:407] Connection header received
0923 20:09:42.247617 (+ 278us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:09:42.247623 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:09:42.247713 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:09:42.247794 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:09:42.251614 (+ 3820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:09:42.252802 (+ 1188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:09:42.273400 (+ 20598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:09:42.274080 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:09:42.274191 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:09:42.288012 (+ 13821us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:09:42.288023 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:09:42.288029 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:09:42.288089 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:09:42.293533 (+ 5444us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:09:42.294526 (+ 993us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:09:42.294534 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:09:42.294539 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:09:42.294708 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:09:42.295168 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:09:42.295174 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:09:42.295177 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:09:42.295525 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:09:42.295656 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0923 20:09:42.297316 (+ 1660us) server_negotiation.cc:299] Negotiation successful
0923 20:09:42.297582 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":385,"thread_start_us":417,"threads_started":1}
I20260923 20:09:42.299137 9137 heartbeater.cc:347] Connected to a master server at 127.1.170.189:40091
I20260923 20:09:42.299299 9137 heartbeater.cc:464] Registering TS with master...
I20260923 20:09:42.299831 9137 heartbeater.cc:511] Master 127.1.170.189:40091 requested a full tablet report, sending...
I20260923 20:09:42.301116 8476 ts_manager.cc:194] Registered new tserver with Master: 0ecb00af331c4c6eb682ec0f64fa0e47 (127.1.170.131:34805)
I20260923 20:09:42.303344 9140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:09:42.253171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41132 (local address 127.1.170.188:36059)
0923 20:09:42.271968 (+ 18797us) server_negotiation.cc:206] Beginning negotiation
0923 20:09:42.271975 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:09:42.280106 (+ 8131us) server_negotiation.cc:407] Connection header received
0923 20:09:42.280426 (+ 320us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:09:42.280434 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:09:42.280578 (+ 144us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:09:42.280656 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:09:42.285571 (+ 4915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:09:42.286969 (+ 1398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:09:42.290464 (+ 3495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:09:42.291128 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:09:42.291234 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:09:42.295594 (+ 4360us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:09:42.295600 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:09:42.295605 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:09:42.295644 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:09:42.299793 (+ 4149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:09:42.300842 (+ 1049us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:09:42.300847 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:09:42.300850 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:09:42.300984 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:09:42.301490 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:09:42.301494 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:09:42.301497 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:09:42.301856 (+ 359us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:09:42.302151 (+ 295us) server_negotiation.cc:1035] Waiting for connection context
0923 20:09:42.302759 (+ 608us) server_negotiation.cc:299] Negotiation successful
0923 20:09:42.302984 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18665,"thread_start_us":37047,"threads_started":1}
I20260923 20:09:42.309345 9141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:09:42.253563 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41880 (local address 127.1.170.190:42111)
0923 20:09:42.273560 (+ 19997us) server_negotiation.cc:206] Beginning negotiation
0923 20:09:42.273569 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 20:09:42.289479 (+ 15910us) server_negotiation.cc:407] Connection header received
0923 20:09:42.291000 (+ 1521us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:09:42.291008 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:09:42.291090 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:09:42.291164 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:09:42.294241 (+ 3077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:09:42.295276 (+ 1035us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:09:42.297166 (+ 1890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:09:42.297826 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:09:42.297970 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:09:42.301639 (+ 3669us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:09:42.301649 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:09:42.301654 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:09:42.301720 (+ 66us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:09:42.306521 (+ 4801us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:09:42.307372 (+ 851us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:09:42.307379 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:09:42.307383 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:09:42.307540 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:09:42.307939 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:09:42.307944 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:09:42.307947 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:09:42.308242 (+ 295us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:09:42.308470 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0923 20:09:42.308795 (+ 325us) server_negotiation.cc:299] Negotiation successful
0923 20:09:42.308994 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19868,"thread_start_us":36739,"threads_started":1}
I20260923 20:09:42.310967 9153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:09:42.254886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45376 (local address 127.1.170.189:40091)
0923 20:09:42.288345 (+ 33459us) server_negotiation.cc:206] Beginning negotiation
0923 20:09:42.288353 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 20:09:42.290388 (+ 2035us) server_negotiation.cc:407] Connection header received
0923 20:09:42.290448 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:09:42.290455 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:09:42.290548 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:09:42.290677 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:09:42.293558 (+ 2881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:09:42.294884 (+ 1326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:09:42.296877 (+ 1993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:09:42.297864 (+ 987us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:09:42.297972 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:09:42.303749 (+ 5777us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:09:42.303755 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:09:42.303760 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:09:42.303808 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:09:42.307726 (+ 3918us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:09:42.308472 (+ 746us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:09:42.308479 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:09:42.308484 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:09:42.308622 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:09:42.309039 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:09:42.309044 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:09:42.309047 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:09:42.309384 (+ 337us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:09:42.309852 (+ 468us) server_negotiation.cc:1035] Waiting for connection context
0923 20:09:42.310381 (+ 529us) server_negotiation.cc:299] Negotiation successful
0923 20:09:42.310601 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33317,"thread_start_us":35483,"threads_started":1}
I20260923 20:09:42.313652 7875 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41880:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 20:09:42.321005 7875 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 20:09:42.362673 9167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:09:42.345771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57910 (local address 127.1.170.130:36095)
0923 20:09:42.350544 (+ 4773us) server_negotiation.cc:206] Beginning negotiation
0923 20:09:42.350558 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0923 20:09:42.350600 (+ 42us) server_negotiation.cc:407] Connection header received
0923 20:09:42.350838 (+ 238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:09:42.350852 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:09:42.351221 (+ 369us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:09:42.351507 (+ 286us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:09:42.352008 (+ 501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:09:42.354704 (+ 2696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:09:42.357783 (+ 3079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:09:42.359860 (+ 2077us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:09:42.360291 (+ 431us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:09:42.360532 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0923 20:09:42.361892 (+ 1360us) server_negotiation.cc:299] Negotiation successful
0923 20:09:42.362198 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4561,"thread_start_us":4384,"threads_started":1}
I20260923 20:09:42.364094 9168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:09:42.345400 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.130:36095 (local address 127.0.0.1:57910)
0923 20:09:42.347087 (+ 1687us) negotiation.cc:107] Waiting for socket to connect
0923 20:09:42.347135 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0923 20:09:42.347275 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:09:42.351492 (+ 4217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:09:42.351496 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:09:42.351520 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:09:42.351831 (+ 311us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:09:42.351841 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:09:42.354882 (+ 3041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:09:42.354888 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:09:42.357510 (+ 2622us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:09:42.357532 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:09:42.360815 (+ 3283us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:09:42.360822 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:09:42.361059 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:09:42.361092 (+ 33us) client_negotiation.cc:769] Sending connection context
0923 20:09:42.361191 (+ 99us) client_negotiation.cc:240] Negotiation successful
0923 20:09:42.363438 (+ 2247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1552,"mutex_wait_us":32,"thread_start_us":6539,"threads_started":1}
I20260923 20:09:42.365511 9169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:09:42.346991 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48590 (local address 127.1.170.129:41229)
0923 20:09:42.347618 (+ 627us) server_negotiation.cc:206] Beginning negotiation
0923 20:09:42.347631 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0923 20:09:42.352393 (+ 4762us) server_negotiation.cc:407] Connection header received
0923 20:09:42.352748 (+ 355us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:09:42.352761 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:09:42.353084 (+ 323us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:09:42.353304 (+ 220us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:09:42.354886 (+ 1582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:09:42.356437 (+ 1551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:09:42.359453 (+ 3016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:09:42.363120 (+ 3667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:09:42.363292 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:09:42.363619 (+ 327us) server_negotiation.cc:1035] Waiting for connection context
0923 20:09:42.364690 (+ 1071us) server_negotiation.cc:299] Negotiation successful
0923 20:09:42.364996 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":411,"thread_start_us":426,"threads_started":1}
I20260923 20:09:42.366561 9166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:09:42.346500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.129:41229 (local address 127.0.0.1:48590)
0923 20:09:42.352278 (+ 5778us) negotiation.cc:107] Waiting for socket to connect
0923 20:09:42.352298 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0923 20:09:42.352503 (+ 205us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:09:42.353617 (+ 1114us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:09:42.353637 (+ 20us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:09:42.353674 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:09:42.354647 (+ 973us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:09:42.354661 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:09:42.356653 (+ 1992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:09:42.356659 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:09:42.359195 (+ 2536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:09:42.359218 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:09:42.364054 (+ 4836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:09:42.364064 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:09:42.364398 (+ 334us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:09:42.364454 (+ 56us) client_negotiation.cc:769] Sending connection context
0923 20:09:42.365783 (+ 1329us) client_negotiation.cc:240] Negotiation successful
0923 20:09:42.366040 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5638,"mutex_wait_us":1,"thread_start_us":746,"threads_started":1}
I20260923 20:09:42.371124 8778 tablet_service.cc:1467] Processing CreateTablet for tablet 11be6fd5d4584413bad4e9055fd5a2a8 (DEFAULT_TABLE table=test-table [id=8c053ef3b3964c96a55a5fcbd0151c29]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 20:09:42.372632 8778 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 11be6fd5d4584413bad4e9055fd5a2a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:09:42.372365 8926 tablet_service.cc:1467] Processing CreateTablet for tablet 11be6fd5d4584413bad4e9055fd5a2a8 (DEFAULT_TABLE table=test-table [id=8c053ef3b3964c96a55a5fcbd0151c29]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 20:09:42.373661 8926 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 11be6fd5d4584413bad4e9055fd5a2a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:09:42.377003 9168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:09:42.352401 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.131:34805 (local address 127.0.0.1:52364)
0923 20:09:42.364620 (+ 12219us) negotiation.cc:107] Waiting for socket to connect
0923 20:09:42.364641 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0923 20:09:42.364902 (+ 261us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:09:42.368954 (+ 4052us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:09:42.368959 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:09:42.368982 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:09:42.369124 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:09:42.369131 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:09:42.371394 (+ 2263us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:09:42.371400 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:09:42.373359 (+ 1959us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:09:42.373385 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:09:42.376136 (+ 2751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:09:42.376143 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:09:42.376329 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:09:42.376365 (+ 36us) client_negotiation.cc:769] Sending connection context
0923 20:09:42.376485 (+ 120us) client_negotiation.cc:240] Negotiation successful
0923 20:09:42.376661 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12074,"thread_start_us":16226,"threads_started":1}
I20260923 20:09:42.377343 9171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:09:42.352866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52364 (local address 127.1.170.131:34805)
0923 20:09:42.367364 (+ 14498us) server_negotiation.cc:206] Beginning negotiation
0923 20:09:42.367377 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0923 20:09:42.367423 (+ 46us) server_negotiation.cc:407] Connection header received
0923 20:09:42.367559 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:09:42.367570 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:09:42.367913 (+ 343us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:09:42.368167 (+ 254us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:09:42.369296 (+ 1129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:09:42.371225 (+ 1929us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:09:42.373560 (+ 2335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:09:42.375611 (+ 2051us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:09:42.375760 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:09:42.376007 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0923 20:09:42.376582 (+ 575us) server_negotiation.cc:299] Negotiation successful
0923 20:09:42.376757 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14259,"thread_start_us":15085,"threads_started":1}
I20260923 20:09:42.381712 9069 tablet_service.cc:1467] Processing CreateTablet for tablet 11be6fd5d4584413bad4e9055fd5a2a8 (DEFAULT_TABLE table=test-table [id=8c053ef3b3964c96a55a5fcbd0151c29]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 20:09:42.383281 9069 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 11be6fd5d4584413bad4e9055fd5a2a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:09:42.387939 9172 tablet_bootstrap.cc:492] T 11be6fd5d4584413bad4e9055fd5a2a8 P e120ba40339e4897a3aa5180c4c45acf: Bootstrap starting.
I20260923 20:09:42.389659 9173 tablet_bootstrap.cc:492] T 11be6fd5d4584413bad4e9055fd5a2a8 P fee8671d00d44381ac37762a73d5ae51: Bootstrap starting.
I20260923 20:09:42.392697 9172 tablet_bootstrap.cc:654] T 11be6fd5d4584413bad4e9055fd5a2a8 P e120ba40339e4897a3aa5180c4c45acf: Neither blocks nor log segments found. Creating new log.
I20260923 20:09:42.393435 9173 tablet_bootstrap.cc:654] T 11be6fd5d4584413bad4e9055fd5a2a8 P fee8671d00d44381ac37762a73d5ae51: Neither blocks nor log segments found. Creating new log.
I20260923 20:09:42.394080 9172 log.cc:824] T 11be6fd5d4584413bad4e9055fd5a2a8 P e120ba40339e4897a3aa5180c4c45acf: Log is configured to *not* fsync() on all Append() calls
I20260923 20:09:42.394662 9173 log.cc:824] T 11be6fd5d4584413bad4e9055fd5a2a8 P fee8671d00d44381ac37762a73d5ae51: Log is configured to *not* fsync() on all Append() calls
I20260923 20:09:42.397581 9172 tablet_bootstrap.cc:492] T 11be6fd5d4584413bad4e9055fd5a2a8 P e120ba40339e4897a3aa5180c4c45acf: No bootstrap required, opened a new log
I20260923 20:09:42.397954 9172 ts_tablet_manager.cc:1397] T 11be6fd5d4584413bad4e9055fd5a2a8 P e120ba40339e4897a3aa5180c4c45acf: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260923 20:09:42.399219 9173 tablet_bootstrap.cc:492] T 11be6fd5d4584413bad4e9055fd5a2a8 P fee8671d00d44381ac37762a73d5ae51: No bootstrap required, opened a new log
I20260923 20:09:42.399533 9173 ts_tablet_manager.cc:1397] T 11be6fd5d4584413bad4e9055fd5a2a8 P fee8671d00d44381ac37762a73d5ae51: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260923 20:09:42.403175 9176 tablet_bootstrap.cc:492] T 11be6fd5d4584413bad4e9055fd5a2a8 P 0ecb00af331c4c6eb682ec0f64fa0e47: Bootstrap starting.
I20260923 20:09:42.407207 9176 tablet_bootstrap.cc:654] T 11be6fd5d4584413bad4e9055fd5a2a8 P 0ecb00af331c4c6eb682ec0f64fa0e47: Neither blocks nor log segments found. Creating new log.
I20260923 20:09:42.408321 9172 raft_consensus.cc:348] T 11be6fd5d4584413bad4e9055fd5a2a8 P e120ba40339e4897a3aa5180c4c45acf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fee8671d00d44381ac37762a73d5ae51" member_type: VOTER last_known_addr { host: "127.1.170.130" port: 36095 } } peers { permanent_uuid: "0ecb00af331c4c6eb682ec0f64fa0e47" member_type: VOTER last_known_addr { host: "127.1.170.131" port: 34805 } } peers { permanent_uuid: "e120ba40339e4897a3aa5180c4c45acf" member_type: VOTER last_known_addr { host: "127.1.170.129" port: 41229 } }
I20260923 20:09:42.408737 9176 log.cc:824] T 11be6fd5d4584413bad4e9055fd5a2a8 P 0ecb00af331c4c6eb682ec0f64fa0e47: Log is configured to *not* fsync() on all Append() calls
I20260923 20:09:42.408795 9172 raft_consensus.cc:374] T 11be6fd5d4584413bad4e9055fd5a2a8 P e120ba40339e4897a3aa5180c4c45acf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:09:42.408895 9172 raft_consensus.cc:729] T 11be6fd5d4584413bad4e9055fd5a2a8 P e120ba40339e4897a3aa5180c4c45acf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e120ba40339e4897a3aa5180c4c45acf, State: Initialized, Role: FOLLOWER
I20260923 20:09:42.409551 9172 consensus_queue.cc:260] T 11be6fd5d4584413bad4e9055fd5a2a8 P e120ba40339e4897a3aa5180c4c45acf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fee8671d00d44381ac37762a73d5ae51" member_type: VOTER last_known_addr { host: "127.1.170.130" port: 36095 } } peers { permanent_uuid: "0ecb00af331c4c6eb682ec0f64fa0e47" member_type: VOTER last_known_addr { host: "127.1.170.131" port: 34805 } } peers { permanent_uuid: "e120ba40339e4897a3aa5180c4c45acf" member_type: VOTER last_known_addr { host: "127.1.170.129" port: 41229 } }
I20260923 20:09:42.411947 9173 raft_consensus.cc:348] T 11be6fd5d4584413bad4e9055fd5a2a8 P fee8671d00d44381ac37762a73d5ae51 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fee8671d00d44381ac37762a73d5ae51" member_type: VOTER last_known_addr { host: "127.1.170.130" port: 36095 } } peers { permanent_uuid: "0ecb00af331c4c6eb682ec0f64fa0e47" member_type: VOTER last_known_addr { host: "127.1.170.131" port: 34805 } } peers { permanent_uuid: "e120ba40339e4897a3aa5180c4c45acf" member_type: VOTER last_known_addr { host: "127.1.170.129" port: 41229 } }
I20260923 20:09:42.412580 9173 raft_consensus.cc:374] T 11be6fd5d4584413bad4e9055fd5a2a8 P fee8671d00d44381ac37762a73d5ae51 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:09:42.412716 9173 raft_consensus.cc:729] T 11be6fd5d4584413bad4e9055fd5a2a8 P fee8671d00d44381ac37762a73d5ae51 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fee8671d00d44381ac37762a73d5ae51, State: Initialized, Role: FOLLOWER
I20260923 20:09:42.413458 9173 consensus_queue.cc:260] T 11be6fd5d4584413bad4e9055fd5a2a8 P fee8671d00d44381ac37762a73d5ae51 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fee8671d00d44381ac37762a73d5ae51" member_type: VOTER last_known_addr { host: "127.1.170.130" port: 36095 } } peers { permanent_uuid: "0ecb00af331c4c6eb682ec0f64fa0e47" member_type: VOTER last_known_addr { host: "127.1.170.131" port: 34805 } } peers { permanent_uuid: "e120ba40339e4897a3aa5180c4c45acf" member_type: VOTER last_known_addr { host: "127.1.170.129" port: 41229 } }
I20260923 20:09:42.415366 9172 ts_tablet_manager.cc:1428] T 11be6fd5d4584413bad4e9055fd5a2a8 P e120ba40339e4897a3aa5180c4c45acf: Time spent starting tablet: real 0.017s user 0.016s sys 0.000s
I20260923 20:09:42.415365 9173 ts_tablet_manager.cc:1428] T 11be6fd5d4584413bad4e9055fd5a2a8 P fee8671d00d44381ac37762a73d5ae51: Time spent starting tablet: real 0.016s user 0.012s sys 0.004s
I20260923 20:09:42.415377 8991 heartbeater.cc:503] Master 127.1.170.190:42111 was elected leader, sending a full tablet report...
I20260923 20:09:42.418466 9176 tablet_bootstrap.cc:492] T 11be6fd5d4584413bad4e9055fd5a2a8 P 0ecb00af331c4c6eb682ec0f64fa0e47: No bootstrap required, opened a new log
I20260923 20:09:42.418771 9176 ts_tablet_manager.cc:1397] T 11be6fd5d4584413bad4e9055fd5a2a8 P 0ecb00af331c4c6eb682ec0f64fa0e47: Time spent bootstrapping tablet: real 0.016s user 0.012s sys 0.000s
I20260923 20:09:42.429481 9176 raft_consensus.cc:348] T 11be6fd5d4584413bad4e9055fd5a2a8 P 0ecb00af331c4c6eb682ec0f64fa0e47 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fee8671d00d44381ac37762a73d5ae51" member_type: VOTER last_known_addr { host: "127.1.170.130" port: 36095 } } peers { permanent_uuid: "0ecb00af331c4c6eb682ec0f64fa0e47" member_type: VOTER last_known_addr { host: "127.1.170.131" port: 34805 } } peers { permanent_uuid: "e120ba40339e4897a3aa5180c4c45acf" member_type: VOTER last_known_addr { host: "127.1.170.129" port: 41229 } }
I20260923 20:09:42.430251 9176 raft_consensus.cc:374] T 11be6fd5d4584413bad4e9055fd5a2a8 P 0ecb00af331c4c6eb682ec0f64fa0e47 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:09:42.430384 9176 raft_consensus.cc:729] T 11be6fd5d4584413bad4e9055fd5a2a8 P 0ecb00af331c4c6eb682ec0f64fa0e47 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ecb00af331c4c6eb682ec0f64fa0e47, State: Initialized, Role: FOLLOWER
I20260923 20:09:42.431267 9176 consensus_queue.cc:260] T 11be6fd5d4584413bad4e9055fd5a2a8 P 0ecb00af331c4c6eb682ec0f64fa0e47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fee8671d00d44381ac37762a73d5ae51" member_type: VOTER last_known_addr { host: "127.1.170.130" port: 36095 } } peers { permanent_uuid: "0ecb00af331c4c6eb682ec0f64fa0e47" member_type: VOTER last_known_addr { host: "127.1.170.131" port: 34805 } } peers { permanent_uuid: "e120ba40339e4897a3aa5180c4c45acf" member_type: VOTER last_known_addr { host: "127.1.170.129" port: 41229 } }
I20260923 20:09:42.432603 9135 heartbeater.cc:503] Master 127.1.170.190:42111 was elected leader, sending a full tablet report...
I20260923 20:09:42.433831 9176 ts_tablet_manager.cc:1428] T 11be6fd5d4584413bad4e9055fd5a2a8 P 0ecb00af331c4c6eb682ec0f64fa0e47: Time spent starting tablet: real 0.015s user 0.014s sys 0.000s
W20260923 20:09:42.440758 8846 tablet.cc:2369] T 11be6fd5d4584413bad4e9055fd5a2a8 P e120ba40339e4897a3aa5180c4c45acf: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260923 20:09:42.492992 9138 tablet.cc:2369] T 11be6fd5d4584413bad4e9055fd5a2a8 P 0ecb00af331c4c6eb682ec0f64fa0e47: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 20:09:42.595631 9179 raft_consensus.cc:482] T 11be6fd5d4584413bad4e9055fd5a2a8 P fee8671d00d44381ac37762a73d5ae51 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 20:09:42.595996 9179 raft_consensus.cc:504] T 11be6fd5d4584413bad4e9055fd5a2a8 P fee8671d00d44381ac37762a73d5ae51 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fee8671d00d44381ac37762a73d5ae51" member_type: VOTER last_known_addr { host: "127.1.170.130" port: 36095 } } peers { permanent_uuid: "0ecb00af331c4c6eb682ec0f64fa0e47" member_type: VOTER last_known_addr { host: "127.1.170.131" port: 34805 } } peers { permanent_uuid: "e120ba40339e4897a3aa5180c4c45acf" member_type: VOTER last_known_addr { host: "127.1.170.129" port: 41229 } }
W20260923 20:09:42.596323 8995 tablet.cc:2369] T 11be6fd5d4584413bad4e9055fd5a2a8 P fee8671d00d44381ac37762a73d5ae51: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 20:09:42.598203 9179 leader_election.cc:288] T 11be6fd5d4584413bad4e9055fd5a2a8 P fee8671d00d44381ac37762a73d5ae51 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0ecb00af331c4c6eb682ec0f64fa0e47 (127.1.170.131:34805), e120ba40339e4897a3aa5180c4c45acf (127.1.170.129:41229)
I20260923 20:09:42.607194 9169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:09:42.598721 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.170.130:37961 (local address 127.1.170.129:41229)
0923 20:09:42.599050 (+ 329us) server_negotiation.cc:206] Beginning negotiation
0923 20:09:42.599060 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0923 20:09:42.599560 (+ 500us) server_negotiation.cc:407] Connection header received
0923 20:09:42.599925 (+ 365us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:09:42.599932 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:09:42.600004 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:09:42.600092 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:09:42.600793 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:09:42.601967 (+ 1174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:09:42.604635 (+ 2668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:09:42.605914 (+ 1279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:09:42.606077 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:09:42.606154 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0923 20:09:42.606817 (+ 663us) server_negotiation.cc:299] Negotiation successful
0923 20:09:42.606942 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128}
I20260923 20:09:42.607192 9171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:09:42.598797 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.170.130:55569 (local address 127.1.170.131:34805)
0923 20:09:42.599031 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0923 20:09:42.599038 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:09:42.599954 (+ 916us) server_negotiation.cc:407] Connection header received
0923 20:09:42.600107 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:09:42.600112 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:09:42.600180 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:09:42.600291 (+ 111us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:09:42.600919 (+ 628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:09:42.601692 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:09:42.604637 (+ 2945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:09:42.605931 (+ 1294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:09:42.606077 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:09:42.606154 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0923 20:09:42.606795 (+ 641us) server_negotiation.cc:299] Negotiation successful
0923 20:09:42.606938 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260923 20:09:42.607425 9182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:09:42.598440 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.131:34805 (local address 127.1.170.130:55569)
0923 20:09:42.599796 (+ 1356us) negotiation.cc:107] Waiting for socket to connect
0923 20:09:42.599816 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0923 20:09:42.599979 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:09:42.600310 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:09:42.600315 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:09:42.600339 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:09:42.600748 (+ 409us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:09:42.600760 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:09:42.601822 (+ 1062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:09:42.601829 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:09:42.604445 (+ 2616us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:09:42.604462 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:09:42.606196 (+ 1734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:09:42.606208 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:09:42.606404 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:09:42.606535 (+ 131us) client_negotiation.cc:769] Sending connection context
0923 20:09:42.606649 (+ 114us) client_negotiation.cc:240] Negotiation successful
0923 20:09:42.606911 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1180,"mutex_wait_us":829,"thread_start_us":390,"threads_started":1}
I20260923 20:09:42.607445 9181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:09:42.598440 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.129:41229 (local address 127.1.170.130:37961)
0923 20:09:42.599447 (+ 1007us) negotiation.cc:107] Waiting for socket to connect
0923 20:09:42.599468 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0923 20:09:42.599684 (+ 216us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:09:42.600121 (+ 437us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:09:42.600125 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:09:42.600145 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:09:42.600611 (+ 466us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:09:42.600624 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:09:42.602108 (+ 1484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:09:42.602113 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:09:42.604445 (+ 2332us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:09:42.604462 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:09:42.606223 (+ 1761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:09:42.606228 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:09:42.606534 (+ 306us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:09:42.606560 (+ 26us) client_negotiation.cc:769] Sending connection context
0923 20:09:42.606725 (+ 165us) client_negotiation.cc:240] Negotiation successful
0923 20:09:42.606911 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":811,"mutex_wait_us":120,"thread_start_us":612,"threads_started":1}
I20260923 20:09:42.608872 9089 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11be6fd5d4584413bad4e9055fd5a2a8" candidate_uuid: "fee8671d00d44381ac37762a73d5ae51" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ecb00af331c4c6eb682ec0f64fa0e47" is_pre_election: true
I20260923 20:09:42.609005 8798 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11be6fd5d4584413bad4e9055fd5a2a8" candidate_uuid: "fee8671d00d44381ac37762a73d5ae51" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e120ba40339e4897a3aa5180c4c45acf" is_pre_election: true
I20260923 20:09:42.609421 9089 raft_consensus.cc:2454] T 11be6fd5d4584413bad4e9055fd5a2a8 P 0ecb00af331c4c6eb682ec0f64fa0e47 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fee8671d00d44381ac37762a73d5ae51 in term 0.
I20260923 20:09:42.609520 8798 raft_consensus.cc:2454] T 11be6fd5d4584413bad4e9055fd5a2a8 P e120ba40339e4897a3aa5180c4c45acf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fee8671d00d44381ac37762a73d5ae51 in term 0.
I20260923 20:09:42.610193 8879 leader_election.cc:302] T 11be6fd5d4584413bad4e9055fd5a2a8 P fee8671d00d44381ac37762a73d5ae51 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 0ecb00af331c4c6eb682ec0f64fa0e47, e120ba40339e4897a3aa5180c4c45acf, fee8671d00d44381ac37762a73d5ae51; no voters:
I20260923 20:09:42.610622 9179 raft_consensus.cc:2781] T 11be6fd5d4584413bad4e9055fd5a2a8 P fee8671d00d44381ac37762a73d5ae51 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 20:09:42.610793 9179 raft_consensus.cc:482] T 11be6fd5d4584413bad4e9055fd5a2a8 P fee8671d00d44381ac37762a73d5ae51 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 20:09:42.610895 9179 raft_consensus.cc:3037] T 11be6fd5d4584413bad4e9055fd5a2a8 P fee8671d00d44381ac37762a73d5ae51 [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:09:42.613188 9179 raft_consensus.cc:504] T 11be6fd5d4584413bad4e9055fd5a2a8 P fee8671d00d44381ac37762a73d5ae51 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fee8671d00d44381ac37762a73d5ae51" member_type: VOTER last_known_addr { host: "127.1.170.130" port: 36095 } } peers { permanent_uuid: "0ecb00af331c4c6eb682ec0f64fa0e47" member_type: VOTER last_known_addr { host: "127.1.170.131" port: 34805 } } peers { permanent_uuid: "e120ba40339e4897a3aa5180c4c45acf" member_type: VOTER last_known_addr { host: "127.1.170.129" port: 41229 } }
I20260923 20:09:42.613979 9179 leader_election.cc:288] T 11be6fd5d4584413bad4e9055fd5a2a8 P fee8671d00d44381ac37762a73d5ae51 [CANDIDATE]: Term 1 election: Requested vote from peers 0ecb00af331c4c6eb682ec0f64fa0e47 (127.1.170.131:34805), e120ba40339e4897a3aa5180c4c45acf (127.1.170.129:41229)
I20260923 20:09:42.614503 9089 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11be6fd5d4584413bad4e9055fd5a2a8" candidate_uuid: "fee8671d00d44381ac37762a73d5ae51" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ecb00af331c4c6eb682ec0f64fa0e47"
I20260923 20:09:42.614547 8798 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11be6fd5d4584413bad4e9055fd5a2a8" candidate_uuid: "fee8671d00d44381ac37762a73d5ae51" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e120ba40339e4897a3aa5180c4c45acf"
I20260923 20:09:42.614734 9089 raft_consensus.cc:3037] T 11be6fd5d4584413bad4e9055fd5a2a8 P 0ecb00af331c4c6eb682ec0f64fa0e47 [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:09:42.614807 8798 raft_consensus.cc:3037] T 11be6fd5d4584413bad4e9055fd5a2a8 P e120ba40339e4897a3aa5180c4c45acf [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:09:42.616955 9089 raft_consensus.cc:2454] T 11be6fd5d4584413bad4e9055fd5a2a8 P 0ecb00af331c4c6eb682ec0f64fa0e47 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fee8671d00d44381ac37762a73d5ae51 in term 1.
I20260923 20:09:42.617015 8798 raft_consensus.cc:2454] T 11be6fd5d4584413bad4e9055fd5a2a8 P e120ba40339e4897a3aa5180c4c45acf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fee8671d00d44381ac37762a73d5ae51 in term 1.
I20260923 20:09:42.617444 8880 leader_election.cc:302] T 11be6fd5d4584413bad4e9055fd5a2a8 P fee8671d00d44381ac37762a73d5ae51 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ecb00af331c4c6eb682ec0f64fa0e47, fee8671d00d44381ac37762a73d5ae51; no voters:
I20260923 20:09:42.617862 9179 raft_consensus.cc:2781] T 11be6fd5d4584413bad4e9055fd5a2a8 P fee8671d00d44381ac37762a73d5ae51 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 20:09:42.618489 9179 raft_consensus.cc:686] T 11be6fd5d4584413bad4e9055fd5a2a8 P fee8671d00d44381ac37762a73d5ae51 [term 1 LEADER]: Becoming Leader. State: Replica: fee8671d00d44381ac37762a73d5ae51, State: Running, Role: LEADER
I20260923 20:09:42.619665 9179 consensus_queue.cc:237] T 11be6fd5d4584413bad4e9055fd5a2a8 P fee8671d00d44381ac37762a73d5ae51 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fee8671d00d44381ac37762a73d5ae51" member_type: VOTER last_known_addr { host: "127.1.170.130" port: 36095 } } peers { permanent_uuid: "0ecb00af331c4c6eb682ec0f64fa0e47" member_type: VOTER last_known_addr { host: "127.1.170.131" port: 34805 } } peers { permanent_uuid: "e120ba40339e4897a3aa5180c4c45acf" member_type: VOTER last_known_addr { host: "127.1.170.129" port: 41229 } }
I20260923 20:09:42.624342 7875 catalog_manager.cc:5495] T 11be6fd5d4584413bad4e9055fd5a2a8 P fee8671d00d44381ac37762a73d5ae51 reported cstate change: term changed from 0 to 1, leader changed from <none> to fee8671d00d44381ac37762a73d5ae51 (127.1.170.130). New cstate: current_term: 1 leader_uuid: "fee8671d00d44381ac37762a73d5ae51" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fee8671d00d44381ac37762a73d5ae51" member_type: VOTER last_known_addr { host: "127.1.170.130" port: 36095 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0ecb00af331c4c6eb682ec0f64fa0e47" member_type: VOTER last_known_addr { host: "127.1.170.131" port: 34805 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e120ba40339e4897a3aa5180c4c45acf" member_type: VOTER last_known_addr { host: "127.1.170.129" port: 41229 } health_report { overall_health: UNKNOWN } } }
I20260923 20:09:42.656127 9167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:09:42.650643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57916 (local address 127.1.170.130:36095)
0923 20:09:42.650890 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0923 20:09:42.650895 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 20:09:42.651025 (+ 130us) server_negotiation.cc:407] Connection header received
0923 20:09:42.651150 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:09:42.651155 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:09:42.651241 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:09:42.651379 (+ 138us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 20:09:42.651762 (+ 383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:09:42.652461 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:09:42.654043 (+ 1582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:09:42.654460 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:09:42.654567 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:09:42.655023 (+ 456us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 20:09:42.655534 (+ 511us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 20:09:42.655666 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0923 20:09:42.655829 (+ 163us) server_negotiation.cc:299] Negotiation successful
0923 20:09:42.655897 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113}
I20260923 20:09:42.666450 8798 raft_consensus.cc:1261] T 11be6fd5d4584413bad4e9055fd5a2a8 P e120ba40339e4897a3aa5180c4c45acf [term 1 FOLLOWER]: Refusing update from remote peer fee8671d00d44381ac37762a73d5ae51: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 20:09:42.666009 9089 raft_consensus.cc:1261] T 11be6fd5d4584413bad4e9055fd5a2a8 P 0ecb00af331c4c6eb682ec0f64fa0e47 [term 1 FOLLOWER]: Refusing update from remote peer fee8671d00d44381ac37762a73d5ae51: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 20:09:42.667682 9183 consensus_queue.cc:1035] T 11be6fd5d4584413bad4e9055fd5a2a8 P fee8671d00d44381ac37762a73d5ae51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e120ba40339e4897a3aa5180c4c45acf" member_type: VOTER last_known_addr { host: "127.1.170.129" port: 41229 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 20:09:42.668159 9179 consensus_queue.cc:1035] T 11be6fd5d4584413bad4e9055fd5a2a8 P fee8671d00d44381ac37762a73d5ae51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ecb00af331c4c6eb682ec0f64fa0e47" member_type: VOTER last_known_addr { host: "127.1.170.131" port: 34805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 20:09:42.690261 9188 mvcc.cc:205] Tried to move back new op lower bound from 7332635372187992064 to 7332635372014260224. Current Snapshot: MvccSnapshot[applied={T|T < 7332635372187992064}]
I20260923 20:09:42.701632 7875 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41880:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 20:09:42.701941 7875 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41880:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 20:09:42.710822 7875 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f83afa4c3fa644f08196bf2ef04d733b: Sending DeleteTablet for 3 replicas of tablet 11be6fd5d4584413bad4e9055fd5a2a8
I20260923 20:09:42.712072 8926 tablet_service.cc:1514] Processing DeleteTablet for tablet 11be6fd5d4584413bad4e9055fd5a2a8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 20:09:42 UTC) from {username='oryx', principal='oryx/127.1.170.190@KRBTEST.COM'} at 127.0.0.1:57910
I20260923 20:09:42.712142 9069 tablet_service.cc:1514] Processing DeleteTablet for tablet 11be6fd5d4584413bad4e9055fd5a2a8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 20:09:42 UTC) from {username='oryx', principal='oryx/127.1.170.190@KRBTEST.COM'} at 127.0.0.1:52364
I20260923 20:09:42.712881 8778 tablet_service.cc:1514] Processing DeleteTablet for tablet 11be6fd5d4584413bad4e9055fd5a2a8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 20:09:42 UTC) from {username='oryx', principal='oryx/127.1.170.190@KRBTEST.COM'} at 127.0.0.1:48590
I20260923 20:09:42.725064 1706 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskqUuku6/build/asan/bin/kudu pid 7843
I20260923 20:09:42.726594 7905 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:09:42.726851 9197 tablet_replica.cc:330] T 11be6fd5d4584413bad4e9055fd5a2a8 P e120ba40339e4897a3aa5180c4c45acf: stopping tablet replica
I20260923 20:09:42.727315 9197 raft_consensus.cc:2229] T 11be6fd5d4584413bad4e9055fd5a2a8 P e120ba40339e4897a3aa5180c4c45acf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 20:09:42.727854 9197 raft_consensus.cc:2258] T 11be6fd5d4584413bad4e9055fd5a2a8 P e120ba40339e4897a3aa5180c4c45acf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 20:09:42.730170 9197 ts_tablet_manager.cc:1905] T 11be6fd5d4584413bad4e9055fd5a2a8 P e120ba40339e4897a3aa5180c4c45acf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 20:09:42.730897 9196 tablet_replica.cc:330] T 11be6fd5d4584413bad4e9055fd5a2a8 P 0ecb00af331c4c6eb682ec0f64fa0e47: stopping tablet replica
I20260923 20:09:42.731551 9196 raft_consensus.cc:2229] T 11be6fd5d4584413bad4e9055fd5a2a8 P 0ecb00af331c4c6eb682ec0f64fa0e47 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 20:09:42.732157 9196 raft_consensus.cc:2258] T 11be6fd5d4584413bad4e9055fd5a2a8 P 0ecb00af331c4c6eb682ec0f64fa0e47 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 20:09:42.733248 9195 tablet_replica.cc:330] T 11be6fd5d4584413bad4e9055fd5a2a8 P fee8671d00d44381ac37762a73d5ae51: stopping tablet replica
I20260923 20:09:42.735723 9196 ts_tablet_manager.cc:1905] T 11be6fd5d4584413bad4e9055fd5a2a8 P 0ecb00af331c4c6eb682ec0f64fa0e47: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 20:09:42.739403 9197 ts_tablet_manager.cc:1918] T 11be6fd5d4584413bad4e9055fd5a2a8 P e120ba40339e4897a3aa5180c4c45acf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 20:09:42.739591 9197 log.cc:1196] T 11be6fd5d4584413bad4e9055fd5a2a8 P e120ba40339e4897a3aa5180c4c45acf: Deleting WAL directory at /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194121577027-1706-0/minicluster-data/ts-0/wal/wals/11be6fd5d4584413bad4e9055fd5a2a8
I20260923 20:09:42.740144 9197 ts_tablet_manager.cc:1939] T 11be6fd5d4584413bad4e9055fd5a2a8 P e120ba40339e4897a3aa5180c4c45acf: Deleting consensus metadata
I20260923 20:09:42.740312 9195 raft_consensus.cc:2229] T 11be6fd5d4584413bad4e9055fd5a2a8 P fee8671d00d44381ac37762a73d5ae51 [term 1 LEADER]: Raft consensus shutting down.
I20260923 20:09:42.741184 9195 raft_consensus.cc:2258] T 11be6fd5d4584413bad4e9055fd5a2a8 P fee8671d00d44381ac37762a73d5ae51 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 20:09:42.743880 9195 ts_tablet_manager.cc:1905] T 11be6fd5d4584413bad4e9055fd5a2a8 P fee8671d00d44381ac37762a73d5ae51: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 20:09:42.751103 9196 ts_tablet_manager.cc:1918] T 11be6fd5d4584413bad4e9055fd5a2a8 P 0ecb00af331c4c6eb682ec0f64fa0e47: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 20:09:42.754067 9196 log.cc:1196] T 11be6fd5d4584413bad4e9055fd5a2a8 P 0ecb00af331c4c6eb682ec0f64fa0e47: Deleting WAL directory at /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194121577027-1706-0/minicluster-data/ts-2/wal/wals/11be6fd5d4584413bad4e9055fd5a2a8
I20260923 20:09:42.754637 9196 ts_tablet_manager.cc:1939] T 11be6fd5d4584413bad4e9055fd5a2a8 P 0ecb00af331c4c6eb682ec0f64fa0e47: Deleting consensus metadata
I20260923 20:09:42.756330 9195 ts_tablet_manager.cc:1918] T 11be6fd5d4584413bad4e9055fd5a2a8 P fee8671d00d44381ac37762a73d5ae51: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 20:09:42.756588 9195 log.cc:1196] T 11be6fd5d4584413bad4e9055fd5a2a8 P fee8671d00d44381ac37762a73d5ae51: Deleting WAL directory at /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194121577027-1706-0/minicluster-data/ts-1/wal/wals/11be6fd5d4584413bad4e9055fd5a2a8
I20260923 20:09:42.757133 9195 ts_tablet_manager.cc:1939] T 11be6fd5d4584413bad4e9055fd5a2a8 P fee8671d00d44381ac37762a73d5ae51: Deleting consensus metadata
I20260923 20:09:42.759353 9153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:09:42.714385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45386 (local address 127.1.170.189:40091)
0923 20:09:42.714701 (+ 316us) server_negotiation.cc:206] Beginning negotiation
0923 20:09:42.714707 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:09:42.722582 (+ 7875us) server_negotiation.cc:407] Connection header received
0923 20:09:42.722928 (+ 346us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:09:42.722934 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:09:42.723091 (+ 157us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:09:42.723171 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:09:42.726280 (+ 3109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:09:42.727433 (+ 1153us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:09:42.733533 (+ 6100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:09:42.734350 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:09:42.734460 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:09:42.740614 (+ 6154us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:09:42.740624 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:09:42.740632 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:09:42.740685 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:09:42.744157 (+ 3472us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:09:42.746150 (+ 1993us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:09:42.746156 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:09:42.746160 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:09:42.746311 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:09:42.746770 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:09:42.746775 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:09:42.746778 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:09:42.747152 (+ 374us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:09:42.747424 (+ 272us) server_negotiation.cc:1035] Waiting for connection context
0923 20:09:42.758727 (+ 11303us) server_negotiation.cc:299] Negotiation successful
0923 20:09:42.759012 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":167}
I20260923 20:09:42.781011 9140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:09:42.722915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41148 (local address 127.1.170.188:36059)
0923 20:09:42.723199 (+ 284us) server_negotiation.cc:206] Beginning negotiation
0923 20:09:42.723204 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 20:09:42.761089 (+ 37885us) server_negotiation.cc:407] Connection header received
0923 20:09:42.762302 (+ 1213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:09:42.762310 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:09:42.762412 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:09:42.762485 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:09:42.764473 (+ 1988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:09:42.765989 (+ 1516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:09:42.768070 (+ 2081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:09:42.768755 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:09:42.768880 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:09:42.773117 (+ 4237us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:09:42.773126 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:09:42.773131 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:09:42.773205 (+ 74us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:09:42.776747 (+ 3542us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:09:42.777606 (+ 859us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:09:42.777612 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:09:42.777616 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:09:42.777770 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:09:42.778263 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:09:42.778283 (+ 20us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:09:42.778287 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:09:42.778653 (+ 366us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:09:42.779148 (+ 495us) server_negotiation.cc:1035] Waiting for connection context
0923 20:09:42.780254 (+ 1106us) server_negotiation.cc:299] Negotiation successful
0923 20:09:42.780504 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":149}
I20260923 20:09:42.878564 7859 catalog_manager.cc:4841] TS e120ba40339e4897a3aa5180c4c45acf (127.1.170.129:41229): tablet 11be6fd5d4584413bad4e9055fd5a2a8 (table test-table [id=8c053ef3b3964c96a55a5fcbd0151c29]) successfully deleted
I20260923 20:09:42.878774 7860 catalog_manager.cc:4841] TS fee8671d00d44381ac37762a73d5ae51 (127.1.170.130:36095): tablet 11be6fd5d4584413bad4e9055fd5a2a8 (table test-table [id=8c053ef3b3964c96a55a5fcbd0151c29]) successfully deleted
I20260923 20:09:42.879037 7860 catalog_manager.cc:4841] TS 0ecb00af331c4c6eb682ec0f64fa0e47 (127.1.170.131:34805): tablet 11be6fd5d4584413bad4e9055fd5a2a8 (table test-table [id=8c053ef3b3964c96a55a5fcbd0151c29]) successfully deleted
I20260923 20:09:42.879274 1706 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqUuku6/build/asan/bin/kudu with pid 7843
I20260923 20:09:42.928795 9153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:09:42.907126 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45390 (local address 127.1.170.189:40091)
0923 20:09:42.907417 (+ 291us) server_negotiation.cc:206] Beginning negotiation
0923 20:09:42.907427 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0923 20:09:42.911513 (+ 4086us) server_negotiation.cc:407] Connection header received
0923 20:09:42.912037 (+ 524us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:09:42.912047 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:09:42.912157 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:09:42.912252 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:09:42.916103 (+ 3851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:09:42.917243 (+ 1140us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:09:42.919486 (+ 2243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:09:42.920004 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:09:42.920145 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:09:42.922898 (+ 2753us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:09:42.922908 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:09:42.922912 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:09:42.922953 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:09:42.926096 (+ 3143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:09:42.926764 (+ 668us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:09:42.926770 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:09:42.926773 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:09:42.926893 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:09:42.927264 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:09:42.927269 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:09:42.927272 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:09:42.927664 (+ 392us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:09:42.927917 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0923 20:09:42.928241 (+ 324us) server_negotiation.cc:299] Negotiation successful
0923 20:09:42.928452 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260923 20:09:42.932972 9140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:09:42.907739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41160 (local address 127.1.170.188:36059)
0923 20:09:42.910102 (+ 2363us) server_negotiation.cc:206] Beginning negotiation
0923 20:09:42.910108 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:09:42.915878 (+ 5770us) server_negotiation.cc:407] Connection header received
0923 20:09:42.916130 (+ 252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:09:42.916136 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:09:42.916222 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:09:42.916291 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:09:42.919283 (+ 2992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:09:42.920463 (+ 1180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:09:42.922919 (+ 2456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:09:42.923553 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:09:42.923714 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:09:42.926768 (+ 3054us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:09:42.926775 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:09:42.926780 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:09:42.926821 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:09:42.930235 (+ 3414us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:09:42.930850 (+ 615us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:09:42.930855 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:09:42.930859 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:09:42.931001 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:09:42.931428 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:09:42.931432 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:09:42.931436 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:09:42.931807 (+ 371us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:09:42.932061 (+ 254us) server_negotiation.cc:1035] Waiting for connection context
0923 20:09:42.932395 (+ 334us) server_negotiation.cc:299] Negotiation successful
0923 20:09:42.932620 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2230}
I20260923 20:09:43.691291 9207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:09:43.690456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.190:42111 (local address 127.1.170.131:53047)
0923 20:09:43.690948 (+ 492us) negotiation.cc:107] Waiting for socket to connect
0923 20:09:43.691120 (+ 172us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.170.190:42111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":367,"thread_start_us":272,"threads_started":1}
W20260923 20:09:43.691686 9135 heartbeater.cc:644] Failed to heartbeat to 127.1.170.190:42111 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.1.170.190:42111: connect: Connection refused (error 111)
I20260923 20:09:43.698256 9208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:09:43.696941 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.190:42111 (local address 127.1.170.130:44805)
0923 20:09:43.697837 (+ 896us) negotiation.cc:107] Waiting for socket to connect
0923 20:09:43.697997 (+ 160us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.170.190:42111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":709,"thread_start_us":590,"threads_started":1}
I20260923 20:09:43.698549 9209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:09:43.697843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.190:42111 (local address 127.1.170.129:37409)
0923 20:09:43.698224 (+ 381us) negotiation.cc:107] Waiting for socket to connect
0923 20:09:43.698349 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.170.190:42111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":289,"thread_start_us":191,"threads_started":1}
W20260923 20:09:43.698637 8991 heartbeater.cc:644] Failed to heartbeat to 127.1.170.190:42111 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.1.170.190:42111: connect: Connection refused (error 111)
W20260923 20:09:43.698864 8843 heartbeater.cc:644] Failed to heartbeat to 127.1.170.190:42111 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.1.170.190:42111: connect: Connection refused (error 111)
I20260923 20:09:44.209743 9210 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0a16f1acbd7c4420b8d8cdf2732a5200 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f83afa4c3fa644f08196bf2ef04d733b)
I20260923 20:09:44.210075 9210 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0a16f1acbd7c4420b8d8cdf2732a5200 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "f83afa4c3fa644f08196bf2ef04d733b" member_type: VOTER last_known_addr { host: "127.1.170.190" port: 42111 } } peers { permanent_uuid: "2fc4ff3aee844aa492e3b79e53ad59c3" member_type: VOTER last_known_addr { host: "127.1.170.189" port: 40091 } attrs { promote: false } } peers { permanent_uuid: "0a16f1acbd7c4420b8d8cdf2732a5200" member_type: VOTER last_known_addr { host: "127.1.170.188" port: 36059 } attrs { promote: false } }
I20260923 20:09:44.210942 9211 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2fc4ff3aee844aa492e3b79e53ad59c3 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f83afa4c3fa644f08196bf2ef04d733b)
I20260923 20:09:44.211193 9211 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2fc4ff3aee844aa492e3b79e53ad59c3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "f83afa4c3fa644f08196bf2ef04d733b" member_type: VOTER last_known_addr { host: "127.1.170.190" port: 42111 } } peers { permanent_uuid: "2fc4ff3aee844aa492e3b79e53ad59c3" member_type: VOTER last_known_addr { host: "127.1.170.189" port: 40091 } attrs { promote: false } } peers { permanent_uuid: "0a16f1acbd7c4420b8d8cdf2732a5200" member_type: VOTER last_known_addr { host: "127.1.170.188" port: 36059 } attrs { promote: false } }
I20260923 20:09:44.211525 9210 leader_election.cc:288] T 00000000000000000000000000000000 P 0a16f1acbd7c4420b8d8cdf2732a5200 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f83afa4c3fa644f08196bf2ef04d733b (127.1.170.190:42111), 2fc4ff3aee844aa492e3b79e53ad59c3 (127.1.170.189:40091)
I20260923 20:09:44.212653 9211 leader_election.cc:288] T 00000000000000000000000000000000 P 2fc4ff3aee844aa492e3b79e53ad59c3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f83afa4c3fa644f08196bf2ef04d733b (127.1.170.190:42111), 0a16f1acbd7c4420b8d8cdf2732a5200 (127.1.170.188:36059)
I20260923 20:09:44.225598 9213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:09:44.211562 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.190:42111 (local address 127.0.0.1:54036)
0923 20:09:44.224593 (+ 13031us) negotiation.cc:107] Waiting for socket to connect
0923 20:09:44.225073 (+ 480us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.170.190:42111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":12877,"thread_start_us":13087,"threads_started":1}
W20260923 20:09:44.226037 8647 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.170.190:42111: connect: Connection refused (error 111)
I20260923 20:09:44.227957 9217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:09:44.225253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.190:42111 (local address 127.0.0.1:54038)
0923 20:09:44.227470 (+ 2217us) negotiation.cc:107] Waiting for socket to connect
0923 20:09:44.227639 (+ 169us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.170.190:42111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":2082,"thread_start_us":1936,"threads_started":1}
W20260923 20:09:44.228410 8460 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.170.190:42111: connect: Connection refused (error 111)
I20260923 20:09:44.228911 9213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:09:44.228498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.190:42111 (local address 127.0.0.1:54052)
0923 20:09:44.228693 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0923 20:09:44.228773 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.170.190:42111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101}
W20260923 20:09:44.229118 8647 leader_election.cc:334] T 00000000000000000000000000000000 P 0a16f1acbd7c4420b8d8cdf2732a5200 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f83afa4c3fa644f08196bf2ef04d733b (127.1.170.190:42111): Network error: Client connection negotiation failed: client connection to 127.1.170.190:42111: connect: Connection refused (error 111)
I20260923 20:09:44.230252 9215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:09:44.212830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.188:36059 (local address 127.0.0.1:43304)
0923 20:09:44.213335 (+ 505us) negotiation.cc:107] Waiting for socket to connect
0923 20:09:44.213358 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0923 20:09:44.213481 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:09:44.219861 (+ 6380us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:09:44.219870 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:09:44.219957 (+ 87us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:09:44.220334 (+ 377us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:09:44.220349 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:09:44.224857 (+ 4508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:09:44.224865 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:09:44.227462 (+ 2597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:09:44.227477 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:09:44.229315 (+ 1838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:09:44.229320 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:09:44.229565 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:09:44.229609 (+ 44us) client_negotiation.cc:769] Sending connection context
0923 20:09:44.229735 (+ 126us) client_negotiation.cc:240] Negotiation successful
0923 20:09:44.229923 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":379,"thread_start_us":603,"threads_started":1}
I20260923 20:09:44.230393 9216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:09:44.213268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43304 (local address 127.1.170.188:36059)
0923 20:09:44.219260 (+ 5992us) server_negotiation.cc:206] Beginning negotiation
0923 20:09:44.219269 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 20:09:44.219296 (+ 27us) server_negotiation.cc:407] Connection header received
0923 20:09:44.219386 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:09:44.219394 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:09:44.219522 (+ 128us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:09:44.220529 (+ 1007us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:09:44.220914 (+ 385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:09:44.224000 (+ 3086us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:09:44.227649 (+ 3649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:09:44.229059 (+ 1410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:09:44.229278 (+ 219us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:09:44.229415 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0923 20:09:44.229772 (+ 357us) server_negotiation.cc:299] Negotiation successful
0923 20:09:44.229871 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":4,"server-negotiator.queue_time_us":5799,"thread_start_us":5657,"threads_started":1}
I20260923 20:09:44.231055 8674 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2fc4ff3aee844aa492e3b79e53ad59c3" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "0a16f1acbd7c4420b8d8cdf2732a5200" is_pre_election: true
I20260923 20:09:44.231305 9215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:09:44.230900 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.190:42111 (local address 127.0.0.1:54064)
0923 20:09:44.231104 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0923 20:09:44.231194 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.170.190:42111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111}
I20260923 20:09:44.231424 8674 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 0a16f1acbd7c4420b8d8cdf2732a5200 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2fc4ff3aee844aa492e3b79e53ad59c3 in term 1.
W20260923 20:09:44.231627 8460 leader_election.cc:334] T 00000000000000000000000000000000 P 2fc4ff3aee844aa492e3b79e53ad59c3 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f83afa4c3fa644f08196bf2ef04d733b (127.1.170.190:42111): Network error: Client connection negotiation failed: client connection to 127.1.170.190:42111: connect: Connection refused (error 111)
I20260923 20:09:44.231962 8462 leader_election.cc:302] T 00000000000000000000000000000000 P 2fc4ff3aee844aa492e3b79e53ad59c3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a16f1acbd7c4420b8d8cdf2732a5200, 2fc4ff3aee844aa492e3b79e53ad59c3; no voters: f83afa4c3fa644f08196bf2ef04d733b
I20260923 20:09:44.232563 9211 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2fc4ff3aee844aa492e3b79e53ad59c3 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260923 20:09:44.232717 9211 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2fc4ff3aee844aa492e3b79e53ad59c3 [term 1 FOLLOWER]: Starting leader election (detected failure of leader f83afa4c3fa644f08196bf2ef04d733b)
I20260923 20:09:44.232856 9211 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2fc4ff3aee844aa492e3b79e53ad59c3 [term 1 FOLLOWER]: Advancing to term 2
I20260923 20:09:44.234865 9214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:09:44.211899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39744 (local address 127.1.170.189:40091)
0923 20:09:44.225203 (+ 13304us) server_negotiation.cc:206] Beginning negotiation
0923 20:09:44.225209 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:09:44.225563 (+ 354us) server_negotiation.cc:407] Connection header received
0923 20:09:44.225633 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:09:44.225638 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:09:44.225894 (+ 256us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:09:44.225991 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:09:44.226990 (+ 999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:09:44.229028 (+ 2038us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:09:44.232360 (+ 3332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:09:44.233721 (+ 1361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:09:44.233868 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:09:44.234025 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0923 20:09:44.234354 (+ 329us) server_negotiation.cc:299] Negotiation successful
0923 20:09:44.234503 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13186,"thread_start_us":12972,"threads_started":1}
I20260923 20:09:44.234861 9212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:09:44.211740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.189:40091 (local address 127.0.0.1:39744)
0923 20:09:44.225311 (+ 13571us) negotiation.cc:107] Waiting for socket to connect
0923 20:09:44.225334 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0923 20:09:44.225458 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:09:44.226144 (+ 686us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:09:44.226148 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:09:44.226173 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:09:44.226761 (+ 588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:09:44.226784 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:09:44.229201 (+ 2417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:09:44.229211 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:09:44.232177 (+ 2966us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:09:44.232192 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:09:44.233965 (+ 1773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:09:44.233973 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:09:44.234167 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:09:44.234207 (+ 40us) client_negotiation.cc:769] Sending connection context
0923 20:09:44.234312 (+ 105us) client_negotiation.cc:240] Negotiation successful
0923 20:09:44.234455 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13413,"mutex_wait_us":732,"thread_start_us":12547,"threads_started":1}
I20260923 20:09:44.236016 8485 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0a16f1acbd7c4420b8d8cdf2732a5200" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "2fc4ff3aee844aa492e3b79e53ad59c3" is_pre_election: true
I20260923 20:09:44.235945 9211 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2fc4ff3aee844aa492e3b79e53ad59c3 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "f83afa4c3fa644f08196bf2ef04d733b" member_type: VOTER last_known_addr { host: "127.1.170.190" port: 42111 } } peers { permanent_uuid: "2fc4ff3aee844aa492e3b79e53ad59c3" member_type: VOTER last_known_addr { host: "127.1.170.189" port: 40091 } attrs { promote: false } } peers { permanent_uuid: "0a16f1acbd7c4420b8d8cdf2732a5200" member_type: VOTER last_known_addr { host: "127.1.170.188" port: 36059 } attrs { promote: false } }
I20260923 20:09:44.236413 8485 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 2fc4ff3aee844aa492e3b79e53ad59c3 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a16f1acbd7c4420b8d8cdf2732a5200 in current term 2: Already voted for candidate 2fc4ff3aee844aa492e3b79e53ad59c3 in this term.
I20260923 20:09:44.236727 9211 leader_election.cc:288] T 00000000000000000000000000000000 P 2fc4ff3aee844aa492e3b79e53ad59c3 [CANDIDATE]: Term 2 election: Requested vote from peers f83afa4c3fa644f08196bf2ef04d733b (127.1.170.190:42111), 0a16f1acbd7c4420b8d8cdf2732a5200 (127.1.170.188:36059)
I20260923 20:09:44.237185 9215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:09:44.236821 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.190:42111 (local address 127.0.0.1:54068)
0923 20:09:44.236953 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0923 20:09:44.237029 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.170.190:42111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260923 20:09:44.237242 8674 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2fc4ff3aee844aa492e3b79e53ad59c3" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "0a16f1acbd7c4420b8d8cdf2732a5200"
I20260923 20:09:44.237489 8674 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0a16f1acbd7c4420b8d8cdf2732a5200 [term 1 FOLLOWER]: Advancing to term 2
I20260923 20:09:44.237807 8650 leader_election.cc:302] T 00000000000000000000000000000000 P 0a16f1acbd7c4420b8d8cdf2732a5200 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a16f1acbd7c4420b8d8cdf2732a5200; no voters: 2fc4ff3aee844aa492e3b79e53ad59c3, f83afa4c3fa644f08196bf2ef04d733b
I20260923 20:09:44.238916 9215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:09:44.238603 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.190:42111 (local address 127.0.0.1:54082)
0923 20:09:44.238738 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0923 20:09:44.238802 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.170.190:42111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260923 20:09:44.239121 8460 leader_election.cc:334] T 00000000000000000000000000000000 P 2fc4ff3aee844aa492e3b79e53ad59c3 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer f83afa4c3fa644f08196bf2ef04d733b (127.1.170.190:42111): Network error: Client connection negotiation failed: client connection to 127.1.170.190:42111: connect: Connection refused (error 111)
I20260923 20:09:44.240132 8674 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 0a16f1acbd7c4420b8d8cdf2732a5200 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2fc4ff3aee844aa492e3b79e53ad59c3 in term 2.
I20260923 20:09:44.240325 9210 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 0a16f1acbd7c4420b8d8cdf2732a5200 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260923 20:09:44.240567 8462 leader_election.cc:302] T 00000000000000000000000000000000 P 2fc4ff3aee844aa492e3b79e53ad59c3 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a16f1acbd7c4420b8d8cdf2732a5200, 2fc4ff3aee844aa492e3b79e53ad59c3; no voters: f83afa4c3fa644f08196bf2ef04d733b
I20260923 20:09:44.240895 9211 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2fc4ff3aee844aa492e3b79e53ad59c3 [term 2 FOLLOWER]: Leader election won for term 2
I20260923 20:09:44.241331 9211 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2fc4ff3aee844aa492e3b79e53ad59c3 [term 2 LEADER]: Becoming Leader. State: Replica: 2fc4ff3aee844aa492e3b79e53ad59c3, State: Running, Role: LEADER
I20260923 20:09:44.241684 9211 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2fc4ff3aee844aa492e3b79e53ad59c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "f83afa4c3fa644f08196bf2ef04d733b" member_type: VOTER last_known_addr { host: "127.1.170.190" port: 42111 } } peers { permanent_uuid: "2fc4ff3aee844aa492e3b79e53ad59c3" member_type: VOTER last_known_addr { host: "127.1.170.189" port: 40091 } attrs { promote: false } } peers { permanent_uuid: "0a16f1acbd7c4420b8d8cdf2732a5200" member_type: VOTER last_known_addr { host: "127.1.170.188" port: 36059 } attrs { promote: false } }
I20260923 20:09:44.243347 9220 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2fc4ff3aee844aa492e3b79e53ad59c3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2fc4ff3aee844aa492e3b79e53ad59c3. Latest consensus state: current_term: 2 leader_uuid: "2fc4ff3aee844aa492e3b79e53ad59c3" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "f83afa4c3fa644f08196bf2ef04d733b" member_type: VOTER last_known_addr { host: "127.1.170.190" port: 42111 } } peers { permanent_uuid: "2fc4ff3aee844aa492e3b79e53ad59c3" member_type: VOTER last_known_addr { host: "127.1.170.189" port: 40091 } attrs { promote: false } } peers { permanent_uuid: "0a16f1acbd7c4420b8d8cdf2732a5200" member_type: VOTER last_known_addr { host: "127.1.170.188" port: 36059 } attrs { promote: false } } }
I20260923 20:09:44.243678 9220 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2fc4ff3aee844aa492e3b79e53ad59c3 [sys.catalog]: This master's current role is: LEADER
I20260923 20:09:44.244374 9222 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 20:09:44.248476 9222 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 20:09:44.249275 9222 catalog_manager.cc:1259] Loaded cluster ID: 62f93adc2448407bb12c561910d92d81
I20260923 20:09:44.249377 9222 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 20:09:44.250584 9222 catalog_manager.cc:1504] Loading token signing keys...
I20260923 20:09:44.251379 9222 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 2fc4ff3aee844aa492e3b79e53ad59c3: Loaded TSK: 0
I20260923 20:09:44.252187 9222 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 20:09:44.264372 8476 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45390:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 20:09:44.271345 8476 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 20:09:44.275859 9215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:09:44.275421 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.190:42111 (local address 127.0.0.1:54100)
0923 20:09:44.275622 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0923 20:09:44.275697 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.170.190:42111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260923 20:09:44.275871 8674 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 0a16f1acbd7c4420b8d8cdf2732a5200 [term 2 FOLLOWER]: Refusing update from remote peer 2fc4ff3aee844aa492e3b79e53ad59c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20260923 20:09:44.276620 9220 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 2fc4ff3aee844aa492e3b79e53ad59c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a16f1acbd7c4420b8d8cdf2732a5200" member_type: VOTER last_known_addr { host: "127.1.170.188" port: 36059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260923 20:09:44.276834 9215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:09:44.276476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.190:42111 (local address 127.0.0.1:54104)
0923 20:09:44.276610 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0923 20:09:44.276680 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.170.190:42111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260923 20:09:44.277065 8460 consensus_peers.cc:487] T 00000000000000000000000000000000 P 2fc4ff3aee844aa492e3b79e53ad59c3 -> Peer f83afa4c3fa644f08196bf2ef04d733b (127.1.170.190:42111): Couldn't send request to peer f83afa4c3fa644f08196bf2ef04d733b. Status: Network error: Client connection negotiation failed: client connection to 127.1.170.190:42111: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260923 20:09:44.279559 9210 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0a16f1acbd7c4420b8d8cdf2732a5200 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2fc4ff3aee844aa492e3b79e53ad59c3. Latest consensus state: current_term: 2 leader_uuid: "2fc4ff3aee844aa492e3b79e53ad59c3" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "f83afa4c3fa644f08196bf2ef04d733b" member_type: VOTER last_known_addr { host: "127.1.170.190" port: 42111 } } peers { permanent_uuid: "2fc4ff3aee844aa492e3b79e53ad59c3" member_type: VOTER last_known_addr { host: "127.1.170.189" port: 40091 } attrs { promote: false } } peers { permanent_uuid: "0a16f1acbd7c4420b8d8cdf2732a5200" member_type: VOTER last_known_addr { host: "127.1.170.188" port: 36059 } attrs { promote: false } } }
I20260923 20:09:44.279850 9210 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0a16f1acbd7c4420b8d8cdf2732a5200 [sys.catalog]: This master's current role is: FOLLOWER
I20260923 20:09:44.281250 9220 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2fc4ff3aee844aa492e3b79e53ad59c3 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "2fc4ff3aee844aa492e3b79e53ad59c3" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "f83afa4c3fa644f08196bf2ef04d733b" member_type: VOTER last_known_addr { host: "127.1.170.190" port: 42111 } } peers { permanent_uuid: "2fc4ff3aee844aa492e3b79e53ad59c3" member_type: VOTER last_known_addr { host: "127.1.170.189" port: 40091 } attrs { promote: false } } peers { permanent_uuid: "0a16f1acbd7c4420b8d8cdf2732a5200" member_type: VOTER last_known_addr { host: "127.1.170.188" port: 36059 } attrs { promote: false } } }
I20260923 20:09:44.281531 9220 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2fc4ff3aee844aa492e3b79e53ad59c3 [sys.catalog]: This master's current role is: LEADER
I20260923 20:09:44.282265 9210 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0a16f1acbd7c4420b8d8cdf2732a5200 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "2fc4ff3aee844aa492e3b79e53ad59c3" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "f83afa4c3fa644f08196bf2ef04d733b" member_type: VOTER last_known_addr { host: "127.1.170.190" port: 42111 } } peers { permanent_uuid: "2fc4ff3aee844aa492e3b79e53ad59c3" member_type: VOTER last_known_addr { host: "127.1.170.189" port: 40091 } attrs { promote: false } } peers { permanent_uuid: "0a16f1acbd7c4420b8d8cdf2732a5200" member_type: VOTER last_known_addr { host: "127.1.170.188" port: 36059 } attrs { promote: false } } }
I20260923 20:09:44.282502 9210 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0a16f1acbd7c4420b8d8cdf2732a5200 [sys.catalog]: This master's current role is: FOLLOWER
I20260923 20:09:44.301815 9217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:09:44.289683 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.130:36095 (local address 127.0.0.1:48050)
0923 20:09:44.289964 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0923 20:09:44.289988 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0923 20:09:44.290101 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:09:44.295647 (+ 5546us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:09:44.295652 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:09:44.295674 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:09:44.295944 (+ 270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:09:44.295954 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:09:44.297211 (+ 1257us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:09:44.297221 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:09:44.299678 (+ 2457us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:09:44.299690 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:09:44.300939 (+ 1249us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:09:44.300948 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:09:44.301152 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:09:44.301198 (+ 46us) client_negotiation.cc:769] Sending connection context
0923 20:09:44.301322 (+ 124us) client_negotiation.cc:240] Negotiation successful
0923 20:09:44.301477 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"mutex_wait_us":51}
I20260923 20:09:44.302117 9229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:09:44.289464 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34956 (local address 127.1.170.131:34805)
0923 20:09:44.296100 (+ 6636us) server_negotiation.cc:206] Beginning negotiation
0923 20:09:44.296104 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 20:09:44.296118 (+ 14us) server_negotiation.cc:407] Connection header received
0923 20:09:44.296178 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:09:44.296181 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:09:44.296235 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:09:44.296344 (+ 109us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:09:44.296723 (+ 379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:09:44.297738 (+ 1015us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:09:44.299651 (+ 1913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:09:44.300944 (+ 1293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:09:44.301071 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:09:44.301175 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0923 20:09:44.301614 (+ 439us) server_negotiation.cc:299] Negotiation successful
0923 20:09:44.301759 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6499,"thread_start_us":6401,"threads_started":1}
I20260923 20:09:44.302322 9215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:09:44.289334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.131:34805 (local address 127.0.0.1:34956)
0923 20:09:44.289804 (+ 470us) negotiation.cc:107] Waiting for socket to connect
0923 20:09:44.289825 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0923 20:09:44.289981 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:09:44.296359 (+ 6378us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:09:44.296363 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:09:44.296386 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:09:44.296574 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:09:44.296583 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:09:44.297933 (+ 1350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:09:44.297939 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:09:44.299507 (+ 1568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:09:44.299518 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:09:44.301085 (+ 1567us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:09:44.301091 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:09:44.301260 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:09:44.301306 (+ 46us) client_negotiation.cc:769] Sending connection context
0923 20:09:44.301378 (+ 72us) client_negotiation.cc:240] Negotiation successful
0923 20:09:44.301509 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":348,"mutex_wait_us":99}
I20260923 20:09:44.302541 9232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:09:44.290532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53064 (local address 127.1.170.129:41229)
0923 20:09:44.293793 (+ 3261us) server_negotiation.cc:206] Beginning negotiation
0923 20:09:44.293802 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 20:09:44.294583 (+ 781us) server_negotiation.cc:407] Connection header received
0923 20:09:44.294675 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:09:44.294683 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:09:44.294796 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:09:44.294934 (+ 138us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:09:44.295722 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:09:44.296944 (+ 1222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:09:44.299846 (+ 2902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:09:44.301109 (+ 1263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:09:44.301237 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:09:44.301363 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0923 20:09:44.302050 (+ 687us) server_negotiation.cc:299] Negotiation successful
0923 20:09:44.302256 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3110,"thread_start_us":3031,"threads_started":1}
I20260923 20:09:44.303082 9230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:09:44.289818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48050 (local address 127.1.170.130:36095)
0923 20:09:44.295393 (+ 5575us) server_negotiation.cc:206] Beginning negotiation
0923 20:09:44.295397 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 20:09:44.295411 (+ 14us) server_negotiation.cc:407] Connection header received
0923 20:09:44.295462 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:09:44.295465 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:09:44.295514 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:09:44.295625 (+ 111us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:09:44.296075 (+ 450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:09:44.297060 (+ 985us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:09:44.299838 (+ 2778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:09:44.300684 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:09:44.300827 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:09:44.300960 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0923 20:09:44.302647 (+ 1687us) server_negotiation.cc:299] Negotiation successful
0923 20:09:44.302800 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5475,"thread_start_us":5394,"threads_started":1}
I20260923 20:09:44.303254 9231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:09:44.290185 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.129:41229 (local address 127.0.0.1:53064)
0923 20:09:44.294276 (+ 4091us) negotiation.cc:107] Waiting for socket to connect
0923 20:09:44.294299 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0923 20:09:44.294453 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:09:44.294966 (+ 513us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:09:44.294974 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:09:44.295012 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:09:44.295494 (+ 482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:09:44.295510 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:09:44.297098 (+ 1588us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:09:44.297106 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:09:44.299574 (+ 2468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:09:44.299589 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:09:44.301525 (+ 1936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:09:44.301535 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:09:44.301727 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:09:44.301768 (+ 41us) client_negotiation.cc:769] Sending connection context
0923 20:09:44.301860 (+ 92us) client_negotiation.cc:240] Negotiation successful
0923 20:09:44.302906 (+ 1046us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3950,"mutex_wait_us":30,"thread_start_us":3809,"threads_started":1}
I20260923 20:09:44.303191 9069 tablet_service.cc:1467] Processing CreateTablet for tablet a81c3b1356944582b68d7e8a109fb493 (DEFAULT_TABLE table=test-table [id=d81fb673489f4da78ce764b526305b62]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 20:09:44.303714 9069 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a81c3b1356944582b68d7e8a109fb493. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:09:44.303745 8926 tablet_service.cc:1467] Processing CreateTablet for tablet a81c3b1356944582b68d7e8a109fb493 (DEFAULT_TABLE table=test-table [id=d81fb673489f4da78ce764b526305b62]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 20:09:44.303985 8778 tablet_service.cc:1467] Processing CreateTablet for tablet a81c3b1356944582b68d7e8a109fb493 (DEFAULT_TABLE table=test-table [id=d81fb673489f4da78ce764b526305b62]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 20:09:44.304149 8926 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a81c3b1356944582b68d7e8a109fb493. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:09:44.304387 8778 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a81c3b1356944582b68d7e8a109fb493. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:09:44.310835 9234 tablet_bootstrap.cc:492] T a81c3b1356944582b68d7e8a109fb493 P fee8671d00d44381ac37762a73d5ae51: Bootstrap starting.
I20260923 20:09:44.311959 9235 tablet_bootstrap.cc:492] T a81c3b1356944582b68d7e8a109fb493 P e120ba40339e4897a3aa5180c4c45acf: Bootstrap starting.
I20260923 20:09:44.313318 9235 tablet_bootstrap.cc:654] T a81c3b1356944582b68d7e8a109fb493 P e120ba40339e4897a3aa5180c4c45acf: Neither blocks nor log segments found. Creating new log.
I20260923 20:09:44.313572 9234 tablet_bootstrap.cc:654] T a81c3b1356944582b68d7e8a109fb493 P fee8671d00d44381ac37762a73d5ae51: Neither blocks nor log segments found. Creating new log.
I20260923 20:09:44.318419 9233 tablet_bootstrap.cc:492] T a81c3b1356944582b68d7e8a109fb493 P 0ecb00af331c4c6eb682ec0f64fa0e47: Bootstrap starting.
I20260923 20:09:44.319262 9235 tablet_bootstrap.cc:492] T a81c3b1356944582b68d7e8a109fb493 P e120ba40339e4897a3aa5180c4c45acf: No bootstrap required, opened a new log
I20260923 20:09:44.319418 9235 ts_tablet_manager.cc:1397] T a81c3b1356944582b68d7e8a109fb493 P e120ba40339e4897a3aa5180c4c45acf: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.003s
I20260923 20:09:44.320243 9235 raft_consensus.cc:348] T a81c3b1356944582b68d7e8a109fb493 P e120ba40339e4897a3aa5180c4c45acf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ecb00af331c4c6eb682ec0f64fa0e47" member_type: VOTER last_known_addr { host: "127.1.170.131" port: 34805 } } peers { permanent_uuid: "fee8671d00d44381ac37762a73d5ae51" member_type: VOTER last_known_addr { host: "127.1.170.130" port: 36095 } } peers { permanent_uuid: "e120ba40339e4897a3aa5180c4c45acf" member_type: VOTER last_known_addr { host: "127.1.170.129" port: 41229 } }
I20260923 20:09:44.320489 9235 raft_consensus.cc:374] T a81c3b1356944582b68d7e8a109fb493 P e120ba40339e4897a3aa5180c4c45acf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:09:44.320580 9235 raft_consensus.cc:729] T a81c3b1356944582b68d7e8a109fb493 P e120ba40339e4897a3aa5180c4c45acf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e120ba40339e4897a3aa5180c4c45acf, State: Initialized, Role: FOLLOWER
I20260923 20:09:44.320840 9233 tablet_bootstrap.cc:654] T a81c3b1356944582b68d7e8a109fb493 P 0ecb00af331c4c6eb682ec0f64fa0e47: Neither blocks nor log segments found. Creating new log.
I20260923 20:09:44.320860 9235 consensus_queue.cc:260] T a81c3b1356944582b68d7e8a109fb493 P e120ba40339e4897a3aa5180c4c45acf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ecb00af331c4c6eb682ec0f64fa0e47" member_type: VOTER last_known_addr { host: "127.1.170.131" port: 34805 } } peers { permanent_uuid: "fee8671d00d44381ac37762a73d5ae51" member_type: VOTER last_known_addr { host: "127.1.170.130" port: 36095 } } peers { permanent_uuid: "e120ba40339e4897a3aa5180c4c45acf" member_type: VOTER last_known_addr { host: "127.1.170.129" port: 41229 } }
I20260923 20:09:44.324036 9234 tablet_bootstrap.cc:492] T a81c3b1356944582b68d7e8a109fb493 P fee8671d00d44381ac37762a73d5ae51: No bootstrap required, opened a new log
I20260923 20:09:44.324180 9234 ts_tablet_manager.cc:1397] T a81c3b1356944582b68d7e8a109fb493 P fee8671d00d44381ac37762a73d5ae51: Time spent bootstrapping tablet: real 0.013s user 0.004s sys 0.008s
I20260923 20:09:44.324203 9233 tablet_bootstrap.cc:492] T a81c3b1356944582b68d7e8a109fb493 P 0ecb00af331c4c6eb682ec0f64fa0e47: No bootstrap required, opened a new log
I20260923 20:09:44.324335 9233 ts_tablet_manager.cc:1397] T a81c3b1356944582b68d7e8a109fb493 P 0ecb00af331c4c6eb682ec0f64fa0e47: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260923 20:09:44.325026 9234 raft_consensus.cc:348] T a81c3b1356944582b68d7e8a109fb493 P fee8671d00d44381ac37762a73d5ae51 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ecb00af331c4c6eb682ec0f64fa0e47" member_type: VOTER last_known_addr { host: "127.1.170.131" port: 34805 } } peers { permanent_uuid: "fee8671d00d44381ac37762a73d5ae51" member_type: VOTER last_known_addr { host: "127.1.170.130" port: 36095 } } peers { permanent_uuid: "e120ba40339e4897a3aa5180c4c45acf" member_type: VOTER last_known_addr { host: "127.1.170.129" port: 41229 } }
I20260923 20:09:44.325248 9234 raft_consensus.cc:374] T a81c3b1356944582b68d7e8a109fb493 P fee8671d00d44381ac37762a73d5ae51 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:09:44.325330 9234 raft_consensus.cc:729] T a81c3b1356944582b68d7e8a109fb493 P fee8671d00d44381ac37762a73d5ae51 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fee8671d00d44381ac37762a73d5ae51, State: Initialized, Role: FOLLOWER
I20260923 20:09:44.325189 9233 raft_consensus.cc:348] T a81c3b1356944582b68d7e8a109fb493 P 0ecb00af331c4c6eb682ec0f64fa0e47 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ecb00af331c4c6eb682ec0f64fa0e47" member_type: VOTER last_known_addr { host: "127.1.170.131" port: 34805 } } peers { permanent_uuid: "fee8671d00d44381ac37762a73d5ae51" member_type: VOTER last_known_addr { host: "127.1.170.130" port: 36095 } } peers { permanent_uuid: "e120ba40339e4897a3aa5180c4c45acf" member_type: VOTER last_known_addr { host: "127.1.170.129" port: 41229 } }
I20260923 20:09:44.325435 9233 raft_consensus.cc:374] T a81c3b1356944582b68d7e8a109fb493 P 0ecb00af331c4c6eb682ec0f64fa0e47 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:09:44.325528 9233 raft_consensus.cc:729] T a81c3b1356944582b68d7e8a109fb493 P 0ecb00af331c4c6eb682ec0f64fa0e47 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ecb00af331c4c6eb682ec0f64fa0e47, State: Initialized, Role: FOLLOWER
I20260923 20:09:44.325574 9234 consensus_queue.cc:260] T a81c3b1356944582b68d7e8a109fb493 P fee8671d00d44381ac37762a73d5ae51 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ecb00af331c4c6eb682ec0f64fa0e47" member_type: VOTER last_known_addr { host: "127.1.170.131" port: 34805 } } peers { permanent_uuid: "fee8671d00d44381ac37762a73d5ae51" member_type: VOTER last_known_addr { host: "127.1.170.130" port: 36095 } } peers { permanent_uuid: "e120ba40339e4897a3aa5180c4c45acf" member_type: VOTER last_known_addr { host: "127.1.170.129" port: 41229 } }
I20260923 20:09:44.325814 9233 consensus_queue.cc:260] T a81c3b1356944582b68d7e8a109fb493 P 0ecb00af331c4c6eb682ec0f64fa0e47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ecb00af331c4c6eb682ec0f64fa0e47" member_type: VOTER last_known_addr { host: "127.1.170.131" port: 34805 } } peers { permanent_uuid: "fee8671d00d44381ac37762a73d5ae51" member_type: VOTER last_known_addr { host: "127.1.170.130" port: 36095 } } peers { permanent_uuid: "e120ba40339e4897a3aa5180c4c45acf" member_type: VOTER last_known_addr { host: "127.1.170.129" port: 41229 } }
I20260923 20:09:44.328040 9233 ts_tablet_manager.cc:1428] T a81c3b1356944582b68d7e8a109fb493 P 0ecb00af331c4c6eb682ec0f64fa0e47: Time spent starting tablet: real 0.004s user 0.003s sys 0.001s
I20260923 20:09:44.329432 9243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:09:44.328363 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.190:42111 (local address 127.1.170.131:53721)
0923 20:09:44.328986 (+ 623us) negotiation.cc:107] Waiting for socket to connect
0923 20:09:44.329148 (+ 162us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.170.190:42111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":457,"thread_start_us":310,"threads_started":1}
I20260923 20:09:44.329439 9234 ts_tablet_manager.cc:1428] T a81c3b1356944582b68d7e8a109fb493 P fee8671d00d44381ac37762a73d5ae51: Time spent starting tablet: real 0.005s user 0.001s sys 0.005s
I20260923 20:09:44.330960 9137 heartbeater.cc:503] Master 127.1.170.189:40091 was elected leader, sending a full tablet report...
I20260923 20:09:44.335996 8994 heartbeater.cc:503] Master 127.1.170.189:40091 was elected leader, sending a full tablet report...
I20260923 20:09:44.338539 9243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:09:44.330132 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.190:42111 (local address 127.1.170.131:50951)
0923 20:09:44.338153 (+ 8021us) negotiation.cc:107] Waiting for socket to connect
0923 20:09:44.338288 (+ 135us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.170.190:42111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":7878,"threads_started":1}
I20260923 20:09:44.338975 9235 ts_tablet_manager.cc:1428] T a81c3b1356944582b68d7e8a109fb493 P e120ba40339e4897a3aa5180c4c45acf: Time spent starting tablet: real 0.019s user 0.001s sys 0.004s
I20260923 20:09:44.339572 8845 heartbeater.cc:503] Master 127.1.170.189:40091 was elected leader, sending a full tablet report...
I20260923 20:09:44.343694 9244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:09:44.329661 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.190:42111 (local address 127.1.170.130:47475)
0923 20:09:44.343408 (+ 13747us) negotiation.cc:107] Waiting for socket to connect
0923 20:09:44.343506 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.170.190:42111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":13645,"threads_started":1}
I20260923 20:09:44.344978 9240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:09:44.323799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.190:42111 (local address 127.1.170.129:38015)
0923 20:09:44.344537 (+ 20738us) negotiation.cc:107] Waiting for socket to connect
0923 20:09:44.344719 (+ 182us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.170.190:42111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":20565,"thread_start_us":20460,"threads_started":1}
I20260923 20:09:44.345187 9244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:09:44.344784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.190:42111 (local address 127.1.170.130:42677)
0923 20:09:44.344930 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0923 20:09:44.345045 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.170.190:42111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260923 20:09:44.346132 9240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:09:44.345777 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.190:42111 (local address 127.1.170.129:45439)
0923 20:09:44.345971 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0923 20:09:44.346034 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.170.190:42111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260923 20:09:44.446563 9242 raft_consensus.cc:482] T a81c3b1356944582b68d7e8a109fb493 P 0ecb00af331c4c6eb682ec0f64fa0e47 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 20:09:44.446830 9242 raft_consensus.cc:504] T a81c3b1356944582b68d7e8a109fb493 P 0ecb00af331c4c6eb682ec0f64fa0e47 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ecb00af331c4c6eb682ec0f64fa0e47" member_type: VOTER last_known_addr { host: "127.1.170.131" port: 34805 } } peers { permanent_uuid: "fee8671d00d44381ac37762a73d5ae51" member_type: VOTER last_known_addr { host: "127.1.170.130" port: 36095 } } peers { permanent_uuid: "e120ba40339e4897a3aa5180c4c45acf" member_type: VOTER last_known_addr { host: "127.1.170.129" port: 41229 } }
I20260923 20:09:44.448400 9242 leader_election.cc:288] T a81c3b1356944582b68d7e8a109fb493 P 0ecb00af331c4c6eb682ec0f64fa0e47 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fee8671d00d44381ac37762a73d5ae51 (127.1.170.130:36095), e120ba40339e4897a3aa5180c4c45acf (127.1.170.129:41229)
I20260923 20:09:44.454070 9230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:09:44.448827 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.170.131:44869 (local address 127.1.170.130:36095)
0923 20:09:44.448997 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0923 20:09:44.449001 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 20:09:44.449018 (+ 17us) server_negotiation.cc:407] Connection header received
0923 20:09:44.449205 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:09:44.449210 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:09:44.449345 (+ 135us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:09:44.449457 (+ 112us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:09:44.449937 (+ 480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:09:44.450622 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:09:44.452308 (+ 1686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:09:44.453085 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:09:44.453311 (+ 226us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:09:44.453389 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0923 20:09:44.453651 (+ 262us) server_negotiation.cc:299] Negotiation successful
0923 20:09:44.453771 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260923 20:09:44.454119 9245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:09:44.448678 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.130:36095 (local address 127.1.170.131:44869)
0923 20:09:44.448882 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0923 20:09:44.448903 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0923 20:09:44.449041 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:09:44.449479 (+ 438us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:09:44.449483 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:09:44.449510 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:09:44.449775 (+ 265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:09:44.449783 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:09:44.450775 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:09:44.450780 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:09:44.452142 (+ 1362us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:09:44.452153 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:09:44.453311 (+ 1158us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:09:44.453317 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:09:44.453490 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:09:44.453517 (+ 27us) client_negotiation.cc:769] Sending connection context
0923 20:09:44.453628 (+ 111us) client_negotiation.cc:240] Negotiation successful
0923 20:09:44.453771 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":56,"mutex_wait_us":1}
I20260923 20:09:44.454269 9232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:09:44.448763 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.170.131:35979 (local address 127.1.170.129:41229)
0923 20:09:44.448950 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0923 20:09:44.448957 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:09:44.449415 (+ 458us) server_negotiation.cc:407] Connection header received
0923 20:09:44.449620 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:09:44.449623 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:09:44.449676 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:09:44.449789 (+ 113us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:09:44.450258 (+ 469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:09:44.450942 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:09:44.452540 (+ 1598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:09:44.453355 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:09:44.453539 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:09:44.453608 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0923 20:09:44.453865 (+ 257us) server_negotiation.cc:299] Negotiation successful
0923 20:09:44.454018 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260923 20:09:44.454519 9243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:09:44.449178 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.129:41229 (local address 127.1.170.131:35979)
0923 20:09:44.449313 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0923 20:09:44.449332 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0923 20:09:44.449474 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:09:44.449806 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:09:44.449811 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:09:44.449831 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:09:44.450120 (+ 289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:09:44.450128 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:09:44.451074 (+ 946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:09:44.451082 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:09:44.452398 (+ 1316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:09:44.452408 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:09:44.453548 (+ 1140us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:09:44.453553 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:09:44.453718 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:09:44.453740 (+ 22us) client_negotiation.cc:769] Sending connection context
0923 20:09:44.453842 (+ 102us) client_negotiation.cc:240] Negotiation successful
0923 20:09:44.454000 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":57,"mutex_wait_us":20}
I20260923 20:09:44.455085 8798 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a81c3b1356944582b68d7e8a109fb493" candidate_uuid: "0ecb00af331c4c6eb682ec0f64fa0e47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e120ba40339e4897a3aa5180c4c45acf" is_pre_election: true
I20260923 20:09:44.455067 8946 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a81c3b1356944582b68d7e8a109fb493" candidate_uuid: "0ecb00af331c4c6eb682ec0f64fa0e47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fee8671d00d44381ac37762a73d5ae51" is_pre_election: true
I20260923 20:09:44.455368 8798 raft_consensus.cc:2454] T a81c3b1356944582b68d7e8a109fb493 P e120ba40339e4897a3aa5180c4c45acf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0ecb00af331c4c6eb682ec0f64fa0e47 in term 0.
I20260923 20:09:44.455448 8946 raft_consensus.cc:2454] T a81c3b1356944582b68d7e8a109fb493 P fee8671d00d44381ac37762a73d5ae51 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0ecb00af331c4c6eb682ec0f64fa0e47 in term 0.
I20260923 20:09:44.455963 9022 leader_election.cc:302] T a81c3b1356944582b68d7e8a109fb493 P 0ecb00af331c4c6eb682ec0f64fa0e47 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 0ecb00af331c4c6eb682ec0f64fa0e47, e120ba40339e4897a3aa5180c4c45acf, fee8671d00d44381ac37762a73d5ae51; no voters:
I20260923 20:09:44.456370 9242 raft_consensus.cc:2781] T a81c3b1356944582b68d7e8a109fb493 P 0ecb00af331c4c6eb682ec0f64fa0e47 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 20:09:44.456465 9242 raft_consensus.cc:482] T a81c3b1356944582b68d7e8a109fb493 P 0ecb00af331c4c6eb682ec0f64fa0e47 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 20:09:44.456574 9242 raft_consensus.cc:3037] T a81c3b1356944582b68d7e8a109fb493 P 0ecb00af331c4c6eb682ec0f64fa0e47 [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:09:44.458832 9242 raft_consensus.cc:504] T a81c3b1356944582b68d7e8a109fb493 P 0ecb00af331c4c6eb682ec0f64fa0e47 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ecb00af331c4c6eb682ec0f64fa0e47" member_type: VOTER last_known_addr { host: "127.1.170.131" port: 34805 } } peers { permanent_uuid: "fee8671d00d44381ac37762a73d5ae51" member_type: VOTER last_known_addr { host: "127.1.170.130" port: 36095 } } peers { permanent_uuid: "e120ba40339e4897a3aa5180c4c45acf" member_type: VOTER last_known_addr { host: "127.1.170.129" port: 41229 } }
I20260923 20:09:44.459452 9242 leader_election.cc:288] T a81c3b1356944582b68d7e8a109fb493 P 0ecb00af331c4c6eb682ec0f64fa0e47 [CANDIDATE]: Term 1 election: Requested vote from peers fee8671d00d44381ac37762a73d5ae51 (127.1.170.130:36095), e120ba40339e4897a3aa5180c4c45acf (127.1.170.129:41229)
I20260923 20:09:44.459841 8946 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a81c3b1356944582b68d7e8a109fb493" candidate_uuid: "0ecb00af331c4c6eb682ec0f64fa0e47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fee8671d00d44381ac37762a73d5ae51"
I20260923 20:09:44.460047 8946 raft_consensus.cc:3037] T a81c3b1356944582b68d7e8a109fb493 P fee8671d00d44381ac37762a73d5ae51 [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:09:44.460086 8798 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a81c3b1356944582b68d7e8a109fb493" candidate_uuid: "0ecb00af331c4c6eb682ec0f64fa0e47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e120ba40339e4897a3aa5180c4c45acf"
I20260923 20:09:44.460338 8798 raft_consensus.cc:3037] T a81c3b1356944582b68d7e8a109fb493 P e120ba40339e4897a3aa5180c4c45acf [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:09:44.462450 8946 raft_consensus.cc:2454] T a81c3b1356944582b68d7e8a109fb493 P fee8671d00d44381ac37762a73d5ae51 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0ecb00af331c4c6eb682ec0f64fa0e47 in term 1.
I20260923 20:09:44.462558 8798 raft_consensus.cc:2454] T a81c3b1356944582b68d7e8a109fb493 P e120ba40339e4897a3aa5180c4c45acf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0ecb00af331c4c6eb682ec0f64fa0e47 in term 1.
I20260923 20:09:44.463012 9023 leader_election.cc:302] T a81c3b1356944582b68d7e8a109fb493 P 0ecb00af331c4c6eb682ec0f64fa0e47 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 0ecb00af331c4c6eb682ec0f64fa0e47, e120ba40339e4897a3aa5180c4c45acf, fee8671d00d44381ac37762a73d5ae51; no voters:
I20260923 20:09:44.463346 9242 raft_consensus.cc:2781] T a81c3b1356944582b68d7e8a109fb493 P 0ecb00af331c4c6eb682ec0f64fa0e47 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 20:09:44.463827 9242 raft_consensus.cc:686] T a81c3b1356944582b68d7e8a109fb493 P 0ecb00af331c4c6eb682ec0f64fa0e47 [term 1 LEADER]: Becoming Leader. State: Replica: 0ecb00af331c4c6eb682ec0f64fa0e47, State: Running, Role: LEADER
I20260923 20:09:44.464437 9242 consensus_queue.cc:237] T a81c3b1356944582b68d7e8a109fb493 P 0ecb00af331c4c6eb682ec0f64fa0e47 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ecb00af331c4c6eb682ec0f64fa0e47" member_type: VOTER last_known_addr { host: "127.1.170.131" port: 34805 } } peers { permanent_uuid: "fee8671d00d44381ac37762a73d5ae51" member_type: VOTER last_known_addr { host: "127.1.170.130" port: 36095 } } peers { permanent_uuid: "e120ba40339e4897a3aa5180c4c45acf" member_type: VOTER last_known_addr { host: "127.1.170.129" port: 41229 } }
I20260923 20:09:44.464823 9243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:09:44.464380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.190:42111 (local address 127.1.170.131:49081)
0923 20:09:44.464543 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0923 20:09:44.464660 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.170.190:42111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260923 20:09:44.468278 8476 catalog_manager.cc:5495] T a81c3b1356944582b68d7e8a109fb493 P 0ecb00af331c4c6eb682ec0f64fa0e47 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0ecb00af331c4c6eb682ec0f64fa0e47 (127.1.170.131). New cstate: current_term: 1 leader_uuid: "0ecb00af331c4c6eb682ec0f64fa0e47" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ecb00af331c4c6eb682ec0f64fa0e47" member_type: VOTER last_known_addr { host: "127.1.170.131" port: 34805 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fee8671d00d44381ac37762a73d5ae51" member_type: VOTER last_known_addr { host: "127.1.170.130" port: 36095 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e120ba40339e4897a3aa5180c4c45acf" member_type: VOTER last_known_addr { host: "127.1.170.129" port: 41229 } health_report { overall_health: UNKNOWN } } }
I20260923 20:09:44.494262 9229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:09:44.489083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34962 (local address 127.1.170.131:34805)
0923 20:09:44.489267 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0923 20:09:44.489271 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 20:09:44.489288 (+ 17us) server_negotiation.cc:407] Connection header received
0923 20:09:44.489482 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:09:44.489487 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:09:44.489562 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:09:44.489687 (+ 125us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 20:09:44.490031 (+ 344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:09:44.490711 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:09:44.491947 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:09:44.492337 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:09:44.492430 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:09:44.492984 (+ 554us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 20:09:44.493543 (+ 559us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 20:09:44.493655 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0923 20:09:44.493847 (+ 192us) server_negotiation.cc:299] Negotiation successful
0923 20:09:44.493988 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260923 20:09:44.498199 8798 raft_consensus.cc:1261] T a81c3b1356944582b68d7e8a109fb493 P e120ba40339e4897a3aa5180c4c45acf [term 1 FOLLOWER]: Refusing update from remote peer 0ecb00af331c4c6eb682ec0f64fa0e47: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 20:09:44.498695 8946 raft_consensus.cc:1261] T a81c3b1356944582b68d7e8a109fb493 P fee8671d00d44381ac37762a73d5ae51 [term 1 FOLLOWER]: Refusing update from remote peer 0ecb00af331c4c6eb682ec0f64fa0e47: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 20:09:44.498834 9246 consensus_queue.cc:1035] T a81c3b1356944582b68d7e8a109fb493 P 0ecb00af331c4c6eb682ec0f64fa0e47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e120ba40339e4897a3aa5180c4c45acf" member_type: VOTER last_known_addr { host: "127.1.170.129" port: 41229 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 20:09:44.499372 9242 consensus_queue.cc:1035] T a81c3b1356944582b68d7e8a109fb493 P 0ecb00af331c4c6eb682ec0f64fa0e47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fee8671d00d44381ac37762a73d5ae51" member_type: VOTER last_known_addr { host: "127.1.170.130" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 20:09:44.501205 9240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:09:44.500640 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.190:42111 (local address 127.1.170.129:54619)
0923 20:09:44.500889 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0923 20:09:44.501003 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.170.190:42111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":138}
I20260923 20:09:44.502635 9244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:09:44.502200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.190:42111 (local address 127.1.170.130:55249)
0923 20:09:44.502407 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0923 20:09:44.502473 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.170.190:42111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102}
I20260923 20:09:44.509414 9249 mvcc.cc:205] Tried to move back new op lower bound from 7332635379697672192 to 7332635379570302976. Current Snapshot: MvccSnapshot[applied={T|T < 7332635379697672192}]
I20260923 20:09:44.512399 9243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:09:44.511864 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.190:42111 (local address 127.1.170.131:51221)
0923 20:09:44.512133 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0923 20:09:44.512221 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.170.190:42111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":141}
I20260923 20:09:44.513166 9240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:09:44.512754 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.190:42111 (local address 127.1.170.129:44733)
0923 20:09:44.512906 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0923 20:09:44.512995 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.170.190:42111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260923 20:09:44.516094 9244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:09:44.515510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.190:42111 (local address 127.1.170.130:45681)
0923 20:09:44.515794 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0923 20:09:44.515901 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.170.190:42111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":137}
I20260923 20:09:44.519289 9243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:09:44.518782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.190:42111 (local address 127.1.170.131:44461)
0923 20:09:44.519049 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0923 20:09:44.519138 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.170.190:42111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":177}
I20260923 20:09:44.527576 8476 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45390:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 20:09:44.527858 8476 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45390:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 20:09:44.533720 8476 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2fc4ff3aee844aa492e3b79e53ad59c3: Sending DeleteTablet for 3 replicas of tablet a81c3b1356944582b68d7e8a109fb493
I20260923 20:09:44.534813 8926 tablet_service.cc:1514] Processing DeleteTablet for tablet a81c3b1356944582b68d7e8a109fb493 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 20:09:44 UTC) from {username='oryx', principal='oryx/127.1.170.189@KRBTEST.COM'} at 127.0.0.1:48050
I20260923 20:09:44.534829 9069 tablet_service.cc:1514] Processing DeleteTablet for tablet a81c3b1356944582b68d7e8a109fb493 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 20:09:44 UTC) from {username='oryx', principal='oryx/127.1.170.189@KRBTEST.COM'} at 127.0.0.1:34956
I20260923 20:09:44.535507 9260 tablet_replica.cc:330] T a81c3b1356944582b68d7e8a109fb493 P 0ecb00af331c4c6eb682ec0f64fa0e47: stopping tablet replica
I20260923 20:09:44.535487 8778 tablet_service.cc:1514] Processing DeleteTablet for tablet a81c3b1356944582b68d7e8a109fb493 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 20:09:44 UTC) from {username='oryx', principal='oryx/127.1.170.189@KRBTEST.COM'} at 127.0.0.1:53064
I20260923 20:09:44.535799 9260 raft_consensus.cc:2229] T a81c3b1356944582b68d7e8a109fb493 P 0ecb00af331c4c6eb682ec0f64fa0e47 [term 1 LEADER]: Raft consensus shutting down.
I20260923 20:09:44.536401 9260 raft_consensus.cc:2258] T a81c3b1356944582b68d7e8a109fb493 P 0ecb00af331c4c6eb682ec0f64fa0e47 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 20:09:44.537488 1706 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskqUuku6/build/asan/bin/kudu pid 8651
I20260923 20:09:44.537907 9260 ts_tablet_manager.cc:1905] T a81c3b1356944582b68d7e8a109fb493 P 0ecb00af331c4c6eb682ec0f64fa0e47: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 20:09:44.546404 9259 tablet_replica.cc:330] T a81c3b1356944582b68d7e8a109fb493 P fee8671d00d44381ac37762a73d5ae51: stopping tablet replica
I20260923 20:09:44.546622 9259 raft_consensus.cc:2229] T a81c3b1356944582b68d7e8a109fb493 P fee8671d00d44381ac37762a73d5ae51 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 20:09:44.546972 9259 raft_consensus.cc:2258] T a81c3b1356944582b68d7e8a109fb493 P fee8671d00d44381ac37762a73d5ae51 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 20:09:44.549471 9261 tablet_replica.cc:330] T a81c3b1356944582b68d7e8a109fb493 P e120ba40339e4897a3aa5180c4c45acf: stopping tablet replica
I20260923 20:09:44.549664 9261 raft_consensus.cc:2229] T a81c3b1356944582b68d7e8a109fb493 P e120ba40339e4897a3aa5180c4c45acf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 20:09:44.549773 9260 ts_tablet_manager.cc:1918] T a81c3b1356944582b68d7e8a109fb493 P 0ecb00af331c4c6eb682ec0f64fa0e47: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 20:09:44.549942 9260 log.cc:1196] T a81c3b1356944582b68d7e8a109fb493 P 0ecb00af331c4c6eb682ec0f64fa0e47: Deleting WAL directory at /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194121577027-1706-0/minicluster-data/ts-2/wal/wals/a81c3b1356944582b68d7e8a109fb493
I20260923 20:09:44.550047 9261 raft_consensus.cc:2258] T a81c3b1356944582b68d7e8a109fb493 P e120ba40339e4897a3aa5180c4c45acf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 20:09:44.550477 9260 ts_tablet_manager.cc:1939] T a81c3b1356944582b68d7e8a109fb493 P 0ecb00af331c4c6eb682ec0f64fa0e47: Deleting consensus metadata
I20260923 20:09:44.551293 9261 ts_tablet_manager.cc:1905] T a81c3b1356944582b68d7e8a109fb493 P e120ba40339e4897a3aa5180c4c45acf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 20:09:44.551697 8461 catalog_manager.cc:4841] TS 0ecb00af331c4c6eb682ec0f64fa0e47 (127.1.170.131:34805): tablet a81c3b1356944582b68d7e8a109fb493 (table test-table [id=d81fb673489f4da78ce764b526305b62]) successfully deleted
I20260923 20:09:44.552259 9259 ts_tablet_manager.cc:1905] T a81c3b1356944582b68d7e8a109fb493 P fee8671d00d44381ac37762a73d5ae51: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 20:09:44.557858 9261 ts_tablet_manager.cc:1918] T a81c3b1356944582b68d7e8a109fb493 P e120ba40339e4897a3aa5180c4c45acf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 20:09:44.558017 9261 log.cc:1196] T a81c3b1356944582b68d7e8a109fb493 P e120ba40339e4897a3aa5180c4c45acf: Deleting WAL directory at /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194121577027-1706-0/minicluster-data/ts-0/wal/wals/a81c3b1356944582b68d7e8a109fb493
I20260923 20:09:44.558465 9261 ts_tablet_manager.cc:1939] T a81c3b1356944582b68d7e8a109fb493 P e120ba40339e4897a3aa5180c4c45acf: Deleting consensus metadata
I20260923 20:09:44.558525 9259 ts_tablet_manager.cc:1918] T a81c3b1356944582b68d7e8a109fb493 P fee8671d00d44381ac37762a73d5ae51: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 20:09:44.558640 9259 log.cc:1196] T a81c3b1356944582b68d7e8a109fb493 P fee8671d00d44381ac37762a73d5ae51: Deleting WAL directory at /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194121577027-1706-0/minicluster-data/ts-1/wal/wals/a81c3b1356944582b68d7e8a109fb493
I20260923 20:09:44.559057 9259 ts_tablet_manager.cc:1939] T a81c3b1356944582b68d7e8a109fb493 P fee8671d00d44381ac37762a73d5ae51: Deleting consensus metadata
I20260923 20:09:44.559731 8460 catalog_manager.cc:4841] TS e120ba40339e4897a3aa5180c4c45acf (127.1.170.129:41229): tablet a81c3b1356944582b68d7e8a109fb493 (table test-table [id=d81fb673489f4da78ce764b526305b62]) successfully deleted
I20260923 20:09:44.560039 8461 catalog_manager.cc:4841] TS fee8671d00d44381ac37762a73d5ae51 (127.1.170.130:36095): tablet a81c3b1356944582b68d7e8a109fb493 (table test-table [id=d81fb673489f4da78ce764b526305b62]) successfully deleted
I20260923 20:09:44.565007 9232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:09:44.539485 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53072 (local address 127.1.170.129:41229)
0923 20:09:44.539755 (+ 270us) server_negotiation.cc:206] Beginning negotiation
0923 20:09:44.539764 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 20:09:44.545538 (+ 5774us) server_negotiation.cc:407] Connection header received
0923 20:09:44.545711 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:09:44.545720 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:09:44.545840 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:09:44.545985 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:09:44.548200 (+ 2215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:09:44.549436 (+ 1236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:09:44.551756 (+ 2320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:09:44.552417 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:09:44.552519 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:09:44.556268 (+ 3749us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:09:44.556291 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:09:44.556310 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:09:44.556368 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:09:44.560469 (+ 4101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:09:44.562759 (+ 2290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:09:44.562775 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:09:44.562778 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:09:44.562911 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:09:44.563310 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:09:44.563313 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:09:44.563315 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:09:44.563709 (+ 394us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:09:44.564128 (+ 419us) server_negotiation.cc:1035] Waiting for connection context
0923 20:09:44.564410 (+ 282us) server_negotiation.cc:299] Negotiation successful
0923 20:09:44.564657 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260923 20:09:44.565677 8838 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:09:44.674696 1706 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqUuku6/build/asan/bin/kudu with pid 8651
I20260923 20:09:44.692176 1706 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskqUuku6/build/asan/bin/kudu pid 8856
I20260923 20:09:44.706497 9230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:09:44.692971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48062 (local address 127.1.170.130:36095)
0923 20:09:44.693154 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0923 20:09:44.693162 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 20:09:44.693188 (+ 26us) server_negotiation.cc:407] Connection header received
0923 20:09:44.693424 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:09:44.693429 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:09:44.693499 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:09:44.693607 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:09:44.694948 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:09:44.695766 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:09:44.697025 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:09:44.697474 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:09:44.697623 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:09:44.700382 (+ 2759us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:09:44.700397 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:09:44.700413 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:09:44.700443 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:09:44.703678 (+ 3235us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:09:44.704337 (+ 659us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:09:44.704352 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:09:44.704355 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:09:44.704474 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:09:44.704869 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:09:44.704876 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:09:44.704879 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:09:44.705244 (+ 365us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:09:44.705552 (+ 308us) server_negotiation.cc:1035] Waiting for connection context
0923 20:09:44.705858 (+ 306us) server_negotiation.cc:299] Negotiation successful
0923 20:09:44.706204 (+ 346us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260923 20:09:44.707132 8986 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:09:44.741643 9231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:09:44.741218 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.190:42111 (local address 127.0.0.1:54112)
0923 20:09:44.741403 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0923 20:09:44.741498 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.170.190:42111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260923 20:09:44.742735 9231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:09:44.742404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.190:42111 (local address 127.0.0.1:54124)
0923 20:09:44.742557 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0923 20:09:44.742636 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.170.190:42111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260923 20:09:44.883365 1706 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqUuku6/build/asan/bin/kudu with pid 8856
I20260923 20:09:44.907346 1706 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskqUuku6/build/asan/bin/kudu pid 9005
I20260923 20:09:44.923928 9229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:09:44.908158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34974 (local address 127.1.170.131:34805)
0923 20:09:44.908340 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0923 20:09:44.908347 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:09:44.908471 (+ 124us) server_negotiation.cc:407] Connection header received
0923 20:09:44.908602 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:09:44.908616 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:09:44.908711 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:09:44.908832 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:09:44.910216 (+ 1384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:09:44.911409 (+ 1193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:09:44.912587 (+ 1178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:09:44.913298 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:09:44.913435 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:09:44.916271 (+ 2836us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:09:44.916291 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:09:44.916319 (+ 28us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:09:44.916367 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:09:44.920640 (+ 4273us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:09:44.921438 (+ 798us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:09:44.921448 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:09:44.921451 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:09:44.921689 (+ 238us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:09:44.922189 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:09:44.922195 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:09:44.922199 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:09:44.922679 (+ 480us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:09:44.922986 (+ 307us) server_negotiation.cc:1035] Waiting for connection context
0923 20:09:44.923290 (+ 304us) server_negotiation.cc:299] Negotiation successful
0923 20:09:44.923514 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260923 20:09:44.924705 9129 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:09:45.058013 1706 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqUuku6/build/asan/bin/kudu with pid 9005
I20260923 20:09:45.078615 1706 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskqUuku6/build/asan/bin/kudu pid 8338
I20260923 20:09:45.079723 8506 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:09:45.201777 1706 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqUuku6/build/asan/bin/kudu with pid 8338
I20260923 20:09:45.224064 1706 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskqUuku6/build/asan/bin/kudu pid 8439
I20260923 20:09:45.225145 8694 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:09:45.341682 1706 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqUuku6/build/asan/bin/kudu with pid 8439
2026-09-23T20:09:45Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (9937 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790194121577027-1706-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 20:09:45 dist-test-slave-th4g krb5kdc[9270](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 20:09:45 dist-test-slave-th4g krb5kdc[9270](info): set up 2 sockets
Sep 23 20:09:45 dist-test-slave-th4g krb5kdc[9270](info): commencing operation
krb5kdc: starting...
W20260923 20:09:46.503216 1706 mini_kdc.cc:121] Time spent starting KDC: real 1.104s user 0.001s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790194121577027-1706-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790194121577027-1706-0/krb5kdc/test-user.keytab.
Sep 23 20:09:46 dist-test-slave-th4g krb5kdc[9270](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194186, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T20:09:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T20:09:46Z Disabled control of system clock
WARNING: no policy specified for oryx/127.1.170.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.1.170.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.1.170.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790194121577027-1706-0/krb5kdc/oryx_127.1.170.190.keytab.
Entry for principal oryx/127.1.170.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790194121577027-1706-0/krb5kdc/oryx_127.1.170.190.keytab.
I20260923 20:09:46.742514 1706 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqUuku6/build/asan/bin/kudu
/tmp/dist-test-taskqUuku6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790194121577027-1706-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790194121577027-1706-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790194121577027-1706-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790194121577027-1706-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.170.190:45507
--webserver_interface=127.1.170.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790194121577027-1706-0/krb5kdc/oryx_127.1.170.190.keytab
--principal=oryx/127.1.170.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.170.148:39783
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.170.190:45507
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790194121577027-1706-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790194121577027-1706-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790194121577027-1706-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:09:47.075508 9283 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:09:47.075807 9283 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:09:47.075968 9283 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:09:47.083426 9283 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 20:09:47.083549 9283 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260923 20:09:47.083590 9283 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:09:47.083710 9283 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:09:47.083818 9283 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 20:09:47.083916 9283 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 20:09:47.094340 9283 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.170.148:39783
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790194121577027-1706-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790194121577027-1706-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.170.190:45507
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790194121577027-1706-0/krb5kdc/oryx_127.1.170.190.keytab
--principal=oryx/127.1.170.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.170.190:45507
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790194121577027-1706-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.170.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790194121577027-1706-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:09:47.095640 9283 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:09:47.097044 9283 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:09:47.109649 9289 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:09:47.109741 9288 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:09:47.109666 9291 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:09:47.110965 9283 server_base.cc:1034] running on GCE node
I20260923 20:09:47.111795 9283 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:09:47.113291 9283 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:09:47.114558 9283 hybrid_clock.cc:648] HybridClock initialized: now 1790194187114484 us; error 44 us; skew 500 ppm
I20260923 20:09:47.114935 9283 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:09:47 dist-test-slave-th4g krb5kdc[9270](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194187, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.170.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:09:47.118275 9283 init.cc:377] Logged in from keytab as oryx/127.1.170.190@KRBTEST.COM (short username oryx)
I20260923 20:09:47.119859 9283 webserver.cc:460] Webserver started at http://127.1.170.190:37567/ using document root <none> and password file <none>
I20260923 20:09:47.120575 9283 fs_manager.cc:359] Metadata directory not provided
I20260923 20:09:47.120705 9283 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:09:47.121053 9283 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:09:47.124347 9283 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790194121577027-1706-0/minicluster-data/master-0/data/instance:
uuid: "6b22557909954a558ccd24b8fbbc4f78"
format_stamp: "Formatted at 2026-09-23 20:09:47 on dist-test-slave-th4g"
server_key: "893fb6b77bcb57ca41600c24e21174eb"
server_key_iv: "6fbaafe0bf714b063c75413b0fdd5258"
server_key_version: "encryptionkey@0"
I20260923 20:09:47.125169 9283 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790194121577027-1706-0/minicluster-data/master-0/wal/instance:
uuid: "6b22557909954a558ccd24b8fbbc4f78"
format_stamp: "Formatted at 2026-09-23 20:09:47 on dist-test-slave-th4g"
server_key: "893fb6b77bcb57ca41600c24e21174eb"
server_key_iv: "6fbaafe0bf714b063c75413b0fdd5258"
server_key_version: "encryptionkey@0"
I20260923 20:09:47.130434 9283 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260923 20:09:47.134151 9298 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:09:47.135574 9283 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260923 20:09:47.135710 9283 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790194121577027-1706-0/minicluster-data/master-0/data,/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790194121577027-1706-0/minicluster-data/master-0/wal
uuid: "6b22557909954a558ccd24b8fbbc4f78"
format_stamp: "Formatted at 2026-09-23 20:09:47 on dist-test-slave-th4g"
server_key: "893fb6b77bcb57ca41600c24e21174eb"
server_key_iv: "6fbaafe0bf714b063c75413b0fdd5258"
server_key_version: "encryptionkey@0"
I20260923 20:09:47.135883 9283 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790194121577027-1706-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790194121577027-1706-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790194121577027-1706-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:09:47.157138 9283 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:09:47.162014 9283 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:09:47.162269 9283 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:09:47.185015 9283 rpc_server.cc:307] RPC server started. Bound to: 127.1.170.190:45507
I20260923 20:09:47.185160 9359 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.170.190:45507 every 8 connection(s)
I20260923 20:09:47.186591 9283 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790194121577027-1706-0/minicluster-data/master-0/data/info.pb
I20260923 20:09:47.190430 9360 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:09:47.195705 1706 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqUuku6/build/asan/bin/kudu as pid 9283
I20260923 20:09:47.195957 1706 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790194121577027-1706-0/minicluster-data/master-0/wal/instance
I20260923 20:09:47.196714 1706 external_mini_cluster.cc:1428] Setting key a3159c9d51e17de06b4a260ec83b5ec1
I20260923 20:09:47.202641 9360 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6b22557909954a558ccd24b8fbbc4f78: Bootstrap starting.
I20260923 20:09:47.206367 9360 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6b22557909954a558ccd24b8fbbc4f78: Neither blocks nor log segments found. Creating new log.
I20260923 20:09:47.207641 9360 log.cc:824] T 00000000000000000000000000000000 P 6b22557909954a558ccd24b8fbbc4f78: Log is configured to *not* fsync() on all Append() calls
I20260923 20:09:47.211375 9360 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6b22557909954a558ccd24b8fbbc4f78: No bootstrap required, opened a new log
Sep 23 20:09:47 dist-test-slave-th4g krb5kdc[9270](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194186, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.1.170.190@KRBTEST.COM
I20260923 20:09:47.219766 9360 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6b22557909954a558ccd24b8fbbc4f78 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b22557909954a558ccd24b8fbbc4f78" member_type: VOTER last_known_addr { host: "127.1.170.190" port: 45507 } }
I20260923 20:09:47.220180 9360 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6b22557909954a558ccd24b8fbbc4f78 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:09:47.220312 9360 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6b22557909954a558ccd24b8fbbc4f78 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b22557909954a558ccd24b8fbbc4f78, State: Initialized, Role: FOLLOWER
I20260923 20:09:47.220940 9360 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6b22557909954a558ccd24b8fbbc4f78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b22557909954a558ccd24b8fbbc4f78" member_type: VOTER last_known_addr { host: "127.1.170.190" port: 45507 } }
I20260923 20:09:47.221222 9360 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6b22557909954a558ccd24b8fbbc4f78 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 20:09:47.221372 9360 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6b22557909954a558ccd24b8fbbc4f78 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 20:09:47.221554 9360 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6b22557909954a558ccd24b8fbbc4f78 [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:09:47.221814 9363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:09:47.198813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50388 (local address 127.1.170.190:45507)
0923 20:09:47.199662 (+ 849us) server_negotiation.cc:206] Beginning negotiation
0923 20:09:47.199674 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0923 20:09:47.199715 (+ 41us) server_negotiation.cc:407] Connection header received
0923 20:09:47.201703 (+ 1988us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:09:47.201727 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:09:47.202231 (+ 504us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:09:47.202591 (+ 360us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:09:47.204331 (+ 1740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:09:47.205772 (+ 1441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:09:47.207635 (+ 1863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:09:47.208843 (+ 1208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:09:47.209066 (+ 223us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:09:47.214008 (+ 4942us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:09:47.214016 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:09:47.214034 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:09:47.214076 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:09:47.217428 (+ 3352us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:09:47.218254 (+ 826us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:09:47.218261 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:09:47.218268 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:09:47.218435 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:09:47.218821 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:09:47.218825 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:09:47.218827 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:09:47.219381 (+ 554us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:09:47.219680 (+ 299us) server_negotiation.cc:1035] Waiting for connection context
0923 20:09:47.220168 (+ 488us) server_negotiation.cc:299] Negotiation successful
0923 20:09:47.220597 (+ 429us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":646,"thread_start_us":406,"threads_started":1}
I20260923 20:09:47.223989 9360 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6b22557909954a558ccd24b8fbbc4f78 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b22557909954a558ccd24b8fbbc4f78" member_type: VOTER last_known_addr { host: "127.1.170.190" port: 45507 } }
I20260923 20:09:47.224628 9360 leader_election.cc:302] T 00000000000000000000000000000000 P 6b22557909954a558ccd24b8fbbc4f78 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6b22557909954a558ccd24b8fbbc4f78; no voters:
I20260923 20:09:47.225353 9360 leader_election.cc:288] T 00000000000000000000000000000000 P 6b22557909954a558ccd24b8fbbc4f78 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 20:09:47.225450 9365 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6b22557909954a558ccd24b8fbbc4f78 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 20:09:47.226259 9365 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6b22557909954a558ccd24b8fbbc4f78 [term 1 LEADER]: Becoming Leader. State: Replica: 6b22557909954a558ccd24b8fbbc4f78, State: Running, Role: LEADER
I20260923 20:09:47.226845 9365 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6b22557909954a558ccd24b8fbbc4f78 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b22557909954a558ccd24b8fbbc4f78" member_type: VOTER last_known_addr { host: "127.1.170.190" port: 45507 } }
I20260923 20:09:47.227674 9360 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6b22557909954a558ccd24b8fbbc4f78 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 20:09:47.229576 9367 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6b22557909954a558ccd24b8fbbc4f78 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6b22557909954a558ccd24b8fbbc4f78. Latest consensus state: current_term: 1 leader_uuid: "6b22557909954a558ccd24b8fbbc4f78" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b22557909954a558ccd24b8fbbc4f78" member_type: VOTER last_known_addr { host: "127.1.170.190" port: 45507 } } }
I20260923 20:09:47.229998 9367 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6b22557909954a558ccd24b8fbbc4f78 [sys.catalog]: This master's current role is: LEADER
I20260923 20:09:47.229586 9366 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6b22557909954a558ccd24b8fbbc4f78 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6b22557909954a558ccd24b8fbbc4f78" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b22557909954a558ccd24b8fbbc4f78" member_type: VOTER last_known_addr { host: "127.1.170.190" port: 45507 } } }
I20260923 20:09:47.230485 9366 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6b22557909954a558ccd24b8fbbc4f78 [sys.catalog]: This master's current role is: LEADER
I20260923 20:09:47.235230 9370 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 20:09:47.241943 9370 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 20:09:47.250566 9370 catalog_manager.cc:1347] Generated new cluster ID: dc6060b22c7d417a9f6718ec11116082
I20260923 20:09:47.250694 9370 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 20:09:47.268620 9370 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 20:09:47.270254 9370 catalog_manager.cc:1504] Loading token signing keys...
I20260923 20:09:47.283095 9370 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6b22557909954a558ccd24b8fbbc4f78: Generated new TSK 0
I20260923 20:09:47.283960 9370 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.1.170.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.1.170.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.1.170.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790194121577027-1706-0/krb5kdc/oryx_127.1.170.129.keytab.
Entry for principal oryx/127.1.170.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790194121577027-1706-0/krb5kdc/oryx_127.1.170.129.keytab.
I20260923 20:09:47.346278 1706 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqUuku6/build/asan/bin/kudu
/tmp/dist-test-taskqUuku6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790194121577027-1706-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790194121577027-1706-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790194121577027-1706-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790194121577027-1706-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.170.129:0
--local_ip_for_outbound_sockets=127.1.170.129
--webserver_interface=127.1.170.129
--webserver_port=0
--tserver_master_addrs=127.1.170.190:45507
--keytab_file=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790194121577027-1706-0/krb5kdc/oryx_127.1.170.129.keytab
--principal=oryx/127.1.170.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.170.148:39783
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790194121577027-1706-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790194121577027-1706-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790194121577027-1706-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:09:47.675333 9386 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:09:47.675669 9386 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:09:47.675846 9386 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:09:47.683311 9386 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:09:47.683480 9386 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:09:47.683634 9386 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260923 20:09:47.683774 9386 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.170.129
I20260923 20:09:47.694175 9386 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.170.148:39783
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790194121577027-1706-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790194121577027-1706-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790194121577027-1706-0/krb5kdc/oryx_127.1.170.129.keytab
--principal=oryx/127.1.170.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.170.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790194121577027-1706-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.170.129
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.1.170.190:45507
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.170.129
--log_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790194121577027-1706-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:09:47.695493 9386 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:09:47.696797 9386 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:09:47.710567 9391 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:09:47.710453 9394 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:09:47.710662 9392 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:09:47.712087 9386 server_base.cc:1034] running on GCE node
I20260923 20:09:47.712769 9386 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:09:47.713757 9386 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:09:47.714972 9386 hybrid_clock.cc:648] HybridClock initialized: now 1790194187714901 us; error 49 us; skew 500 ppm
I20260923 20:09:47.715274 9386 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:09:47 dist-test-slave-th4g krb5kdc[9270](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194187, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.170.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:09:47.718447 9386 init.cc:377] Logged in from keytab as oryx/127.1.170.129@KRBTEST.COM (short username oryx)
I20260923 20:09:47.719894 9386 webserver.cc:460] Webserver started at http://127.1.170.129:36281/ using document root <none> and password file <none>
I20260923 20:09:47.720532 9386 fs_manager.cc:359] Metadata directory not provided
I20260923 20:09:47.720614 9386 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:09:47.721000 9386 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:09:47.724455 9386 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790194121577027-1706-0/minicluster-data/ts-0/data/instance:
uuid: "c8c21250d251462898a337ffe611517e"
format_stamp: "Formatted at 2026-09-23 20:09:47 on dist-test-slave-th4g"
server_key: "89f209343cf303246bddc88cfb5860e4"
server_key_iv: "94a392da69c9edfa581b25eeb23d9c65"
server_key_version: "encryptionkey@0"
I20260923 20:09:47.725279 9386 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790194121577027-1706-0/minicluster-data/ts-0/wal/instance:
uuid: "c8c21250d251462898a337ffe611517e"
format_stamp: "Formatted at 2026-09-23 20:09:47 on dist-test-slave-th4g"
server_key: "89f209343cf303246bddc88cfb5860e4"
server_key_iv: "94a392da69c9edfa581b25eeb23d9c65"
server_key_version: "encryptionkey@0"
I20260923 20:09:47.730947 9386 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.005s
I20260923 20:09:47.734462 9401 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:09:47.735944 9386 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260923 20:09:47.736124 9386 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790194121577027-1706-0/minicluster-data/ts-0/data,/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790194121577027-1706-0/minicluster-data/ts-0/wal
uuid: "c8c21250d251462898a337ffe611517e"
format_stamp: "Formatted at 2026-09-23 20:09:47 on dist-test-slave-th4g"
server_key: "89f209343cf303246bddc88cfb5860e4"
server_key_iv: "94a392da69c9edfa581b25eeb23d9c65"
server_key_version: "encryptionkey@0"
I20260923 20:09:47.736358 9386 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790194121577027-1706-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790194121577027-1706-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790194121577027-1706-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:09:47.786666 9386 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:09:47.791079 9386 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:09:47.791357 9386 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:09:47.794075 9386 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 20:09:47.794152 9386 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 20:09:47.794333 9386 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 20:09:47.794413 9386 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 23 20:09:47 dist-test-slave-th4g krb5kdc[9270](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194187, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.170.129@KRBTEST.COM for oryx/127.1.170.190@KRBTEST.COM
I20260923 20:09:47.820611 9410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:09:47.795104 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.170.129:52949 (local address 127.1.170.190:45507)
0923 20:09:47.797534 (+ 2430us) server_negotiation.cc:206] Beginning negotiation
0923 20:09:47.797541 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:09:47.797566 (+ 25us) server_negotiation.cc:407] Connection header received
0923 20:09:47.799524 (+ 1958us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:09:47.799530 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:09:47.799629 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:09:47.799767 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:09:47.802540 (+ 2773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:09:47.803387 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:09:47.807166 (+ 3779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:09:47.807651 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:09:47.807832 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:09:47.812474 (+ 4642us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:09:47.812481 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:09:47.812484 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:09:47.812518 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:09:47.816343 (+ 3825us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:09:47.817634 (+ 1291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:09:47.817643 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:09:47.817648 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:09:47.817801 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:09:47.818435 (+ 634us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:09:47.818440 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:09:47.818444 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:09:47.818760 (+ 316us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:09:47.818955 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0923 20:09:47.820090 (+ 1135us) server_negotiation.cc:299] Negotiation successful
0923 20:09:47.820311 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2304,"thread_start_us":2209,"threads_started":1}
I20260923 20:09:47.822589 9411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:09:47.795743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.190:45507 (local address 127.1.170.129:52949)
0923 20:09:47.796933 (+ 1190us) negotiation.cc:107] Waiting for socket to connect
0923 20:09:47.797004 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0923 20:09:47.799227 (+ 2223us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:09:47.800197 (+ 970us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:09:47.800214 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:09:47.801198 (+ 984us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:09:47.802294 (+ 1096us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:09:47.802325 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:09:47.803613 (+ 1288us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:09:47.803622 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:09:47.806971 (+ 3349us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:09:47.806991 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:09:47.807948 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:09:47.807960 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:09:47.808357 (+ 397us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:09:47.809374 (+ 1017us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:09:47.809397 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:09:47.812185 (+ 2788us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:09:47.816540 (+ 4355us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:09:47.816554 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:09:47.816580 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:09:47.817461 (+ 881us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:09:47.817969 (+ 508us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:09:47.817981 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:09:47.817992 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:09:47.818294 (+ 302us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:09:47.818924 (+ 630us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:09:47.818936 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:09:47.819560 (+ 624us) client_negotiation.cc:769] Sending connection context
0923 20:09:47.819967 (+ 407us) client_negotiation.cc:240] Negotiation successful
0923 20:09:47.820687 (+ 720us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":724,"thread_start_us":455,"threads_started":1}
I20260923 20:09:47.845202 9476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:09:47.829034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.190:45507 (local address 127.1.170.129:36723)
0923 20:09:47.829674 (+ 640us) negotiation.cc:107] Waiting for socket to connect
0923 20:09:47.829691 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0923 20:09:47.829868 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:09:47.830280 (+ 412us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:09:47.830285 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:09:47.830909 (+ 624us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:09:47.831636 (+ 727us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:09:47.831658 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:09:47.833242 (+ 1584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:09:47.833250 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:09:47.835283 (+ 2033us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:09:47.835299 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:09:47.836130 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:09:47.836136 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:09:47.836327 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:09:47.837052 (+ 725us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:09:47.837070 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:09:47.837943 (+ 873us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:09:47.842071 (+ 4128us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:09:47.842080 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:09:47.842085 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:09:47.842814 (+ 729us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:09:47.843313 (+ 499us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:09:47.843319 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:09:47.843323 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:09:47.843554 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:09:47.844290 (+ 736us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:09:47.844294 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:09:47.844452 (+ 158us) client_negotiation.cc:769] Sending connection context
0923 20:09:47.844565 (+ 113us) client_negotiation.cc:240] Negotiation successful
0923 20:09:47.844833 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":512,"thread_start_us":361,"threads_started":1}
I20260923 20:09:47.845512 9410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:09:47.828962 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.170.129:36723 (local address 127.1.170.190:45507)
0923 20:09:47.829124 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0923 20:09:47.829129 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 20:09:47.829790 (+ 661us) server_negotiation.cc:407] Connection header received
0923 20:09:47.830071 (+ 281us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:09:47.830075 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:09:47.830138 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:09:47.830251 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:09:47.831802 (+ 1551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:09:47.832982 (+ 1180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:09:47.835446 (+ 2464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:09:47.835925 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:09:47.836084 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:09:47.838237 (+ 2153us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:09:47.838245 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:09:47.838252 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:09:47.838296 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:09:47.841923 (+ 3627us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:09:47.842969 (+ 1046us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:09:47.842978 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:09:47.842983 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:09:47.843159 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:09:47.843708 (+ 549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:09:47.843716 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:09:47.843722 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:09:47.844150 (+ 428us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:09:47.844347 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0923 20:09:47.844744 (+ 397us) server_negotiation.cc:299] Negotiation successful
0923 20:09:47.845034 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260923 20:09:47.851136 9386 rpc_server.cc:307] RPC server started. Bound to: 127.1.170.129:34387
I20260923 20:09:47.851174 9521 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.170.129:34387 every 8 connection(s)
I20260923 20:09:47.852268 9386 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790194121577027-1706-0/minicluster-data/ts-0/data/info.pb
I20260923 20:09:47.854318 9522 heartbeater.cc:347] Connected to a master server at 127.1.170.190:45507
I20260923 20:09:47.854554 9522 heartbeater.cc:464] Registering TS with master...
I20260923 20:09:47.855172 9522 heartbeater.cc:511] Master 127.1.170.190:45507 requested a full tablet report, sending...
I20260923 20:09:47.857280 9315 ts_manager.cc:194] Registered new tserver with Master: c8c21250d251462898a337ffe611517e (127.1.170.129:34387)
I20260923 20:09:47.858841 9315 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.1.170.129@KRBTEST.COM'} at 127.1.170.129:52949
I20260923 20:09:47.861436 1706 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqUuku6/build/asan/bin/kudu as pid 9386
I20260923 20:09:47.861658 1706 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790194121577027-1706-0/minicluster-data/ts-0/wal/instance
I20260923 20:09:47.862352 1706 external_mini_cluster.cc:1428] Setting key a3d8231e16d9290e41f7e2a6d1724ace
WARNING: no policy specified for oryx/127.1.170.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.1.170.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.1.170.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790194121577027-1706-0/krb5kdc/oryx_127.1.170.130.keytab.
Entry for principal oryx/127.1.170.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790194121577027-1706-0/krb5kdc/oryx_127.1.170.130.keytab.
I20260923 20:09:47.910698 1706 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqUuku6/build/asan/bin/kudu
/tmp/dist-test-taskqUuku6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790194121577027-1706-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790194121577027-1706-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790194121577027-1706-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790194121577027-1706-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.170.130:0
--local_ip_for_outbound_sockets=127.1.170.130
--webserver_interface=127.1.170.130
--webserver_port=0
--tserver_master_addrs=127.1.170.190:45507
--keytab_file=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790194121577027-1706-0/krb5kdc/oryx_127.1.170.130.keytab
--principal=oryx/127.1.170.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.170.148:39783
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790194121577027-1706-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790194121577027-1706-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790194121577027-1706-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:09:48.234185 9526 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:09:48.234510 9526 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:09:48.234679 9526 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:09:48.242197 9526 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:09:48.242355 9526 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:09:48.242499 9526 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260923 20:09:48.242640 9526 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.170.130
I20260923 20:09:48.252444 9526 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.170.148:39783
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790194121577027-1706-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790194121577027-1706-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790194121577027-1706-0/krb5kdc/oryx_127.1.170.130.keytab
--principal=oryx/127.1.170.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.170.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790194121577027-1706-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.170.130
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.1.170.190:45507
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.170.130
--log_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790194121577027-1706-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:09:48.253764 9526 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:09:48.255088 9526 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:09:48.268203 9532 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:09:48.268278 9531 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:09:48.268252 9534 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:09:48.269104 9526 server_base.cc:1034] running on GCE node
I20260923 20:09:48.269641 9526 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:09:48.270671 9526 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:09:48.271878 9526 hybrid_clock.cc:648] HybridClock initialized: now 1790194188271806 us; error 52 us; skew 500 ppm
I20260923 20:09:48.272125 9526 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:09:48 dist-test-slave-th4g krb5kdc[9270](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194188, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.170.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:09:48.275240 9526 init.cc:377] Logged in from keytab as oryx/127.1.170.130@KRBTEST.COM (short username oryx)
I20260923 20:09:48.276608 9526 webserver.cc:460] Webserver started at http://127.1.170.130:41679/ using document root <none> and password file <none>
I20260923 20:09:48.277266 9526 fs_manager.cc:359] Metadata directory not provided
I20260923 20:09:48.277350 9526 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:09:48.277604 9526 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:09:48.280539 9526 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790194121577027-1706-0/minicluster-data/ts-1/data/instance:
uuid: "f7f31863969a4bda9e128166c15ac1ad"
format_stamp: "Formatted at 2026-09-23 20:09:48 on dist-test-slave-th4g"
server_key: "fdd49faee281ec0fedfc622129739de1"
server_key_iv: "671e2c6b992b51a85126d1f30d137ec6"
server_key_version: "encryptionkey@0"
I20260923 20:09:48.281311 9526 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790194121577027-1706-0/minicluster-data/ts-1/wal/instance:
uuid: "f7f31863969a4bda9e128166c15ac1ad"
format_stamp: "Formatted at 2026-09-23 20:09:48 on dist-test-slave-th4g"
server_key: "fdd49faee281ec0fedfc622129739de1"
server_key_iv: "671e2c6b992b51a85126d1f30d137ec6"
server_key_version: "encryptionkey@0"
I20260923 20:09:48.286564 9526 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.003s
I20260923 20:09:48.290079 9541 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:09:48.291621 9526 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260923 20:09:48.291777 9526 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790194121577027-1706-0/minicluster-data/ts-1/data,/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790194121577027-1706-0/minicluster-data/ts-1/wal
uuid: "f7f31863969a4bda9e128166c15ac1ad"
format_stamp: "Formatted at 2026-09-23 20:09:48 on dist-test-slave-th4g"
server_key: "fdd49faee281ec0fedfc622129739de1"
server_key_iv: "671e2c6b992b51a85126d1f30d137ec6"
server_key_version: "encryptionkey@0"
I20260923 20:09:48.291962 9526 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790194121577027-1706-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790194121577027-1706-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790194121577027-1706-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:09:48.306545 9526 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:09:48.310885 9526 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:09:48.311199 9526 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:09:48.314143 9526 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 20:09:48.314227 9526 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 20:09:48.314415 9526 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 20:09:48.314476 9526 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 23 20:09:48 dist-test-slave-th4g krb5kdc[9270](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194188, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.170.130@KRBTEST.COM for oryx/127.1.170.190@KRBTEST.COM
I20260923 20:09:48.340615 9410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:09:48.314860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.170.130:38455 (local address 127.1.170.190:45507)
0923 20:09:48.315068 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0923 20:09:48.315077 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 20:09:48.316551 (+ 1474us) server_negotiation.cc:407] Connection header received
0923 20:09:48.319911 (+ 3360us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:09:48.319920 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:09:48.320011 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:09:48.320147 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:09:48.322988 (+ 2841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:09:48.324408 (+ 1420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:09:48.327080 (+ 2672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:09:48.327763 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:09:48.327911 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:09:48.332574 (+ 4663us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:09:48.332578 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:09:48.332582 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:09:48.332614 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:09:48.336173 (+ 3559us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:09:48.337286 (+ 1113us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:09:48.337295 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:09:48.337301 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:09:48.337469 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:09:48.338159 (+ 690us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:09:48.338166 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:09:48.338171 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:09:48.338556 (+ 385us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:09:48.338733 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0923 20:09:48.339897 (+ 1164us) server_negotiation.cc:299] Negotiation successful
0923 20:09:48.340174 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260923 20:09:48.342514 9550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:09:48.315416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.190:45507 (local address 127.1.170.130:38455)
0923 20:09:48.316364 (+ 948us) negotiation.cc:107] Waiting for socket to connect
0923 20:09:48.316429 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0923 20:09:48.319507 (+ 3078us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:09:48.320589 (+ 1082us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:09:48.320604 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:09:48.321455 (+ 851us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:09:48.322640 (+ 1185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:09:48.322663 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:09:48.324603 (+ 1940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:09:48.324607 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:09:48.326917 (+ 2310us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:09:48.326931 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:09:48.328027 (+ 1096us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:09:48.328035 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:09:48.328343 (+ 308us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:09:48.329281 (+ 938us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:09:48.329321 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:09:48.332337 (+ 3016us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:09:48.336358 (+ 4021us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:09:48.336370 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:09:48.336396 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:09:48.337130 (+ 734us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:09:48.337625 (+ 495us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:09:48.337631 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:09:48.337637 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:09:48.337999 (+ 362us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:09:48.338711 (+ 712us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:09:48.338719 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:09:48.339308 (+ 589us) client_negotiation.cc:769] Sending connection context
0923 20:09:48.339738 (+ 430us) client_negotiation.cc:240] Negotiation successful
0923 20:09:48.340452 (+ 714us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":613,"thread_start_us":368,"threads_started":1}
I20260923 20:09:48.358227 9526 rpc_server.cc:307] RPC server started. Bound to: 127.1.170.130:35035
I20260923 20:09:48.358294 9657 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.170.130:35035 every 8 connection(s)
I20260923 20:09:48.359633 9526 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790194121577027-1706-0/minicluster-data/ts-1/data/info.pb
I20260923 20:09:48.361831 9660 heartbeater.cc:347] Connected to a master server at 127.1.170.190:45507
I20260923 20:09:48.362108 9660 heartbeater.cc:464] Registering TS with master...
I20260923 20:09:48.362309 1706 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqUuku6/build/asan/bin/kudu as pid 9526
I20260923 20:09:48.362473 1706 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790194121577027-1706-0/minicluster-data/ts-1/wal/instance
I20260923 20:09:48.362954 9660 heartbeater.cc:511] Master 127.1.170.190:45507 requested a full tablet report, sending...
I20260923 20:09:48.363037 1706 external_mini_cluster.cc:1428] Setting key d7feb584c8abc625c7d6480b0359b7cb
I20260923 20:09:48.364650 9314 ts_manager.cc:194] Registered new tserver with Master: f7f31863969a4bda9e128166c15ac1ad (127.1.170.130:35035)
I20260923 20:09:48.365852 9314 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.1.170.130@KRBTEST.COM'} at 127.1.170.130:38455
I20260923 20:09:48.375293 9410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:09:48.360594 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.170.130:37715 (local address 127.1.170.190:45507)
0923 20:09:48.360803 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0923 20:09:48.360811 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 20:09:48.362130 (+ 1319us) server_negotiation.cc:407] Connection header received
0923 20:09:48.362319 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:09:48.362325 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:09:48.362420 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:09:48.362533 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:09:48.363834 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:09:48.364788 (+ 954us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:09:48.366535 (+ 1747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:09:48.367294 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:09:48.367436 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:09:48.369316 (+ 1880us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:09:48.369325 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:09:48.369333 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:09:48.369377 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:09:48.372566 (+ 3189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:09:48.373418 (+ 852us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:09:48.373422 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:09:48.373424 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:09:48.373537 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:09:48.373970 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:09:48.373975 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:09:48.373978 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:09:48.374290 (+ 312us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:09:48.374399 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0923 20:09:48.374903 (+ 504us) server_negotiation.cc:299] Negotiation successful
0923 20:09:48.375064 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260923 20:09:48.375689 9662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:09:48.360875 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.190:45507 (local address 127.1.170.130:37715)
0923 20:09:48.362006 (+ 1131us) negotiation.cc:107] Waiting for socket to connect
0923 20:09:48.362028 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0923 20:09:48.362185 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:09:48.362585 (+ 400us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:09:48.362589 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:09:48.363220 (+ 631us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:09:48.363594 (+ 374us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:09:48.363610 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:09:48.364894 (+ 1284us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:09:48.364901 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:09:48.366393 (+ 1492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:09:48.366406 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:09:48.367491 (+ 1085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:09:48.367497 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:09:48.367672 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:09:48.368384 (+ 712us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:09:48.368406 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:09:48.369126 (+ 720us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:09:48.372707 (+ 3581us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:09:48.372714 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:09:48.372721 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:09:48.373308 (+ 587us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:09:48.373640 (+ 332us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:09:48.373648 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:09:48.373654 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:09:48.373824 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:09:48.374404 (+ 580us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:09:48.374412 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:09:48.374624 (+ 212us) client_negotiation.cc:769] Sending connection context
0923 20:09:48.374806 (+ 182us) client_negotiation.cc:240] Negotiation successful
0923 20:09:48.375146 (+ 340us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":939,"thread_start_us":646,"threads_started":1}
WARNING: no policy specified for oryx/127.1.170.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.1.170.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.1.170.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790194121577027-1706-0/krb5kdc/oryx_127.1.170.131.keytab.
Entry for principal oryx/127.1.170.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790194121577027-1706-0/krb5kdc/oryx_127.1.170.131.keytab.
I20260923 20:09:48.413712 1706 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqUuku6/build/asan/bin/kudu
/tmp/dist-test-taskqUuku6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790194121577027-1706-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790194121577027-1706-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790194121577027-1706-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790194121577027-1706-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.170.131:0
--local_ip_for_outbound_sockets=127.1.170.131
--webserver_interface=127.1.170.131
--webserver_port=0
--tserver_master_addrs=127.1.170.190:45507
--keytab_file=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790194121577027-1706-0/krb5kdc/oryx_127.1.170.131.keytab
--principal=oryx/127.1.170.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.170.148:39783
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790194121577027-1706-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790194121577027-1706-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790194121577027-1706-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:09:48.732832 9665 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:09:48.733120 9665 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:09:48.733279 9665 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:09:48.741134 9665 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:09:48.741267 9665 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:09:48.741426 9665 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260923 20:09:48.741544 9665 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.170.131
I20260923 20:09:48.751298 9665 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.170.148:39783
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790194121577027-1706-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790194121577027-1706-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790194121577027-1706-0/krb5kdc/oryx_127.1.170.131.keytab
--principal=oryx/127.1.170.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.170.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790194121577027-1706-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.170.131
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.1.170.190:45507
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.170.131
--log_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790194121577027-1706-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:09:48.752511 9665 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:09:48.753832 9665 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:09:48.766767 9670 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:09:48.766812 9671 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:09:48.766853 9673 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:09:48.767577 9665 server_base.cc:1034] running on GCE node
I20260923 20:09:48.768203 9665 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:09:48.769120 9665 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:09:48.770367 9665 hybrid_clock.cc:648] HybridClock initialized: now 1790194188770304 us; error 41 us; skew 500 ppm
I20260923 20:09:48.770690 9665 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:09:48 dist-test-slave-th4g krb5kdc[9270](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194188, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.170.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:09:48.773849 9665 init.cc:377] Logged in from keytab as oryx/127.1.170.131@KRBTEST.COM (short username oryx)
I20260923 20:09:48.775269 9665 webserver.cc:460] Webserver started at http://127.1.170.131:38291/ using document root <none> and password file <none>
I20260923 20:09:48.775959 9665 fs_manager.cc:359] Metadata directory not provided
I20260923 20:09:48.776088 9665 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:09:48.776410 9665 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:09:48.779342 9665 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790194121577027-1706-0/minicluster-data/ts-2/data/instance:
uuid: "3176a36c35d2480fb87fa57226b3bd7d"
format_stamp: "Formatted at 2026-09-23 20:09:48 on dist-test-slave-th4g"
server_key: "a1da4c40adc3ebcc3668b19daa909162"
server_key_iv: "82389bf9f387ff9c7ed94c32d757f688"
server_key_version: "encryptionkey@0"
I20260923 20:09:48.780133 9665 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790194121577027-1706-0/minicluster-data/ts-2/wal/instance:
uuid: "3176a36c35d2480fb87fa57226b3bd7d"
format_stamp: "Formatted at 2026-09-23 20:09:48 on dist-test-slave-th4g"
server_key: "a1da4c40adc3ebcc3668b19daa909162"
server_key_iv: "82389bf9f387ff9c7ed94c32d757f688"
server_key_version: "encryptionkey@0"
I20260923 20:09:48.785390 9665 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260923 20:09:48.788968 9680 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:09:48.790458 9665 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 20:09:48.790637 9665 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790194121577027-1706-0/minicluster-data/ts-2/data,/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790194121577027-1706-0/minicluster-data/ts-2/wal
uuid: "3176a36c35d2480fb87fa57226b3bd7d"
format_stamp: "Formatted at 2026-09-23 20:09:48 on dist-test-slave-th4g"
server_key: "a1da4c40adc3ebcc3668b19daa909162"
server_key_iv: "82389bf9f387ff9c7ed94c32d757f688"
server_key_version: "encryptionkey@0"
I20260923 20:09:48.790871 9665 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790194121577027-1706-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790194121577027-1706-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790194121577027-1706-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:09:48.808547 9665 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:09:48.812793 9665 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:09:48.813077 9665 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:09:48.815865 9665 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 20:09:48.815979 9665 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 20:09:48.816119 9665 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 20:09:48.816191 9665 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 23 20:09:48 dist-test-slave-th4g krb5kdc[9270](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194188, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.170.131@KRBTEST.COM for oryx/127.1.170.190@KRBTEST.COM
I20260923 20:09:48.838107 9410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:09:48.816206 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.170.131:38663 (local address 127.1.170.190:45507)
0923 20:09:48.816428 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0923 20:09:48.816435 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:09:48.817822 (+ 1387us) server_negotiation.cc:407] Connection header received
0923 20:09:48.819745 (+ 1923us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:09:48.819750 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:09:48.819842 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:09:48.819951 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:09:48.822118 (+ 2167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:09:48.823160 (+ 1042us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:09:48.825765 (+ 2605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:09:48.826374 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:09:48.826510 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:09:48.831393 (+ 4883us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:09:48.831400 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:09:48.831404 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:09:48.831453 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:09:48.835071 (+ 3618us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:09:48.835808 (+ 737us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:09:48.835813 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:09:48.835815 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:09:48.835944 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:09:48.836389 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:09:48.836396 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:09:48.836405 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:09:48.836653 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:09:48.836809 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0923 20:09:48.837496 (+ 687us) server_negotiation.cc:299] Negotiation successful
0923 20:09:48.837711 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260923 20:09:48.839134 9689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:09:48.816770 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.190:45507 (local address 127.1.170.131:38663)
0923 20:09:48.817625 (+ 855us) negotiation.cc:107] Waiting for socket to connect
0923 20:09:48.817675 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0923 20:09:48.819477 (+ 1802us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:09:48.820190 (+ 713us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:09:48.820200 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:09:48.820924 (+ 724us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:09:48.821918 (+ 994us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:09:48.821948 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:09:48.823321 (+ 1373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:09:48.823327 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:09:48.825607 (+ 2280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:09:48.825621 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:09:48.826614 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:09:48.826621 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:09:48.826988 (+ 367us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:09:48.827840 (+ 852us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:09:48.827874 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:09:48.831160 (+ 3286us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:09:48.835214 (+ 4054us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:09:48.835224 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:09:48.835240 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:09:48.835664 (+ 424us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:09:48.836055 (+ 391us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:09:48.836061 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:09:48.836065 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:09:48.836273 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:09:48.836783 (+ 510us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:09:48.836791 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:09:48.837131 (+ 340us) client_negotiation.cc:769] Sending connection context
0923 20:09:48.837386 (+ 255us) client_negotiation.cc:240] Negotiation successful
0923 20:09:48.837929 (+ 543us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":594,"thread_start_us":402,"threads_started":1}
I20260923 20:09:48.861109 9410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:09:48.845085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.170.131:58377 (local address 127.1.170.190:45507)
0923 20:09:48.845453 (+ 368us) server_negotiation.cc:206] Beginning negotiation
0923 20:09:48.845461 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 20:09:48.846060 (+ 599us) server_negotiation.cc:407] Connection header received
0923 20:09:48.846318 (+ 258us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:09:48.846326 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:09:48.846421 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:09:48.846543 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:09:48.848006 (+ 1463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:09:48.849058 (+ 1052us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:09:48.851053 (+ 1995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:09:48.851935 (+ 882us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:09:48.852103 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:09:48.854435 (+ 2332us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:09:48.854441 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:09:48.854446 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:09:48.854486 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:09:48.857961 (+ 3475us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:09:48.858901 (+ 940us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:09:48.858907 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:09:48.858910 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:09:48.859050 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:09:48.859615 (+ 565us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:09:48.859622 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:09:48.859628 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:09:48.859950 (+ 322us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:09:48.860113 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0923 20:09:48.860512 (+ 399us) server_negotiation.cc:299] Negotiation successful
0923 20:09:48.860733 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":232}
I20260923 20:09:48.861241 9741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:09:48.845113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.190:45507 (local address 127.1.170.131:58377)
0923 20:09:48.845931 (+ 818us) negotiation.cc:107] Waiting for socket to connect
0923 20:09:48.845955 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0923 20:09:48.846138 (+ 183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:09:48.846571 (+ 433us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:09:48.846576 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:09:48.847202 (+ 626us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:09:48.847859 (+ 657us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:09:48.847878 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:09:48.849204 (+ 1326us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:09:48.849209 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:09:48.850887 (+ 1678us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:09:48.850903 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:09:48.852149 (+ 1246us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:09:48.852164 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:09:48.852372 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:09:48.853165 (+ 793us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:09:48.853187 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:09:48.854206 (+ 1019us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:09:48.858142 (+ 3936us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:09:48.858153 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:09:48.858169 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:09:48.858772 (+ 603us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:09:48.859229 (+ 457us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:09:48.859237 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:09:48.859245 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:09:48.859436 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:09:48.860072 (+ 636us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:09:48.860077 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:09:48.860265 (+ 188us) client_negotiation.cc:769] Sending connection context
0923 20:09:48.860408 (+ 143us) client_negotiation.cc:240] Negotiation successful
0923 20:09:48.860795 (+ 387us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":609,"thread_start_us":366,"threads_started":1}
I20260923 20:09:48.861764 9522 heartbeater.cc:503] Master 127.1.170.190:45507 was elected leader, sending a full tablet report...
I20260923 20:09:48.876886 9665 rpc_server.cc:307] RPC server started. Bound to: 127.1.170.131:45757
I20260923 20:09:48.876915 9799 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.170.131:45757 every 8 connection(s)
I20260923 20:09:48.878201 9665 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790194121577027-1706-0/minicluster-data/ts-2/data/info.pb
I20260923 20:09:48.880654 9800 heartbeater.cc:347] Connected to a master server at 127.1.170.190:45507
I20260923 20:09:48.880874 9800 heartbeater.cc:464] Registering TS with master...
I20260923 20:09:48.881502 9800 heartbeater.cc:511] Master 127.1.170.190:45507 requested a full tablet report, sending...
I20260923 20:09:48.882870 9315 ts_manager.cc:194] Registered new tserver with Master: 3176a36c35d2480fb87fa57226b3bd7d (127.1.170.131:45757)
I20260923 20:09:48.883801 9315 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.1.170.131@KRBTEST.COM'} at 127.1.170.131:38663
I20260923 20:09:48.884845 1706 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqUuku6/build/asan/bin/kudu as pid 9665
I20260923 20:09:48.885035 1706 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790194121577027-1706-0/minicluster-data/ts-2/wal/instance
I20260923 20:09:48.885524 1706 external_mini_cluster.cc:1428] Setting key 8bf0666a87e9c1e61c429bb780babb48
I20260923 20:09:48.888608 1706 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 23 20:09:48 dist-test-slave-th4g krb5kdc[9270](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194188, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 23 20:09:48 dist-test-slave-th4g krb5kdc[9270](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.1.170.190@KRBTEST.COM, Server not found in Kerberos database
Sep 23 20:09:48 dist-test-slave-th4g krb5kdc[9270](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.1.170.190@KRBTEST.COM, Server not found in Kerberos database
I20260923 20:09:49.368786 9660 heartbeater.cc:503] Master 127.1.170.190:45507 was elected leader, sending a full tablet report...
I20260923 20:09:49.886919 9800 heartbeater.cc:503] Master 127.1.170.190:45507 was elected leader, sending a full tablet report...
W20260923 20:09:50.034965 9808 sasl_common.cc:323] Time spent calling sasl_client_start(): real 1.100s user 0.000s sys 0.015s
W20260923 20:09:50.036145 9410 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 20:09:48.929388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50396 (local address 127.1.170.190:45507)
0923 20:09:48.929624 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0923 20:09:48.929631 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:09:48.929807 (+ 176us) server_negotiation.cc:407] Connection header received
0923 20:09:48.930199 (+ 392us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:09:48.930204 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:09:48.930271 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:09:48.930381 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:09:48.931693 (+ 1312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:09:48.932428 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:09:48.933680 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:09:48.934169 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:09:48.934316 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:09:50.035878 (+1101562us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50396: BlockingRecv error: recv got EOF from 127.0.0.1:50396 (error 108)
Metrics: {"server-negotiator.queue_time_us":81}
Sep 23 20:09:50 dist-test-slave-th4g krb5kdc[9270](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194188, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.1.170.190@KRBTEST.COM
I20260923 20:09:50.054312 9410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:09:50.039990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50408 (local address 127.1.170.190:45507)
0923 20:09:50.040246 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0923 20:09:50.040254 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 20:09:50.040547 (+ 293us) server_negotiation.cc:407] Connection header received
0923 20:09:50.040758 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:09:50.040764 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:09:50.040873 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:09:50.041001 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:09:50.042218 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:09:50.042948 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:09:50.044187 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:09:50.044685 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:09:50.044822 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:09:50.048876 (+ 4054us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:09:50.048881 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:09:50.048885 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:09:50.048920 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:09:50.051783 (+ 2863us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:09:50.052526 (+ 743us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:09:50.052530 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:09:50.052533 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:09:50.052681 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:09:50.053062 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:09:50.053065 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:09:50.053067 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:09:50.053366 (+ 299us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:09:50.053582 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0923 20:09:50.053818 (+ 236us) server_negotiation.cc:299] Negotiation successful
0923 20:09:50.054086 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260923 20:09:50.059119 9315 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50408:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 20:09:50.066339 9315 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 20:09:50.092823 9822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:09:50.079933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59158 (local address 127.1.170.131:45757)
0923 20:09:50.080690 (+ 757us) server_negotiation.cc:206] Beginning negotiation
0923 20:09:50.080701 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0923 20:09:50.081001 (+ 300us) server_negotiation.cc:407] Connection header received
0923 20:09:50.083017 (+ 2016us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:09:50.083029 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:09:50.083293 (+ 264us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:09:50.083724 (+ 431us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:09:50.084417 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:09:50.086047 (+ 1630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:09:50.090047 (+ 4000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:09:50.091588 (+ 1541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:09:50.091750 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:09:50.091963 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0923 20:09:50.092313 (+ 350us) server_negotiation.cc:299] Negotiation successful
0923 20:09:50.092503 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":584,"thread_start_us":415,"threads_started":1}
I20260923 20:09:50.092880 9820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:09:50.079280 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.131:45757 (local address 127.0.0.1:59158)
0923 20:09:50.080848 (+ 1568us) negotiation.cc:107] Waiting for socket to connect
0923 20:09:50.080873 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0923 20:09:50.082033 (+ 1160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:09:50.083546 (+ 1513us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:09:50.083554 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:09:50.083593 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:09:50.084175 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:09:50.084194 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:09:50.086361 (+ 2167us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:09:50.086370 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:09:50.089794 (+ 3424us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:09:50.089810 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:09:50.091831 (+ 2021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:09:50.091840 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:09:50.092044 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:09:50.092102 (+ 58us) client_negotiation.cc:769] Sending connection context
0923 20:09:50.092242 (+ 140us) client_negotiation.cc:240] Negotiation successful
0923 20:09:50.092446 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1389,"mutex_wait_us":225,"spinlock_wait_cycles":9344,"thread_start_us":857,"threads_started":1}
I20260923 20:09:50.094206 9821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:09:50.079088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.129:34387 (local address 127.0.0.1:47884)
0923 20:09:50.080426 (+ 1338us) negotiation.cc:107] Waiting for socket to connect
0923 20:09:50.080475 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0923 20:09:50.082034 (+ 1559us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:09:50.083183 (+ 1149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:09:50.083195 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:09:50.083250 (+ 55us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:09:50.084563 (+ 1313us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:09:50.084578 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:09:50.087274 (+ 2696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:09:50.087283 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:09:50.090858 (+ 3575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:09:50.090872 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:09:50.092838 (+ 1966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:09:50.092847 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:09:50.093460 (+ 613us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:09:50.093495 (+ 35us) client_negotiation.cc:769] Sending connection context
0923 20:09:50.093587 (+ 92us) client_negotiation.cc:240] Negotiation successful
0923 20:09:50.093737 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":989,"mutex_wait_us":2,"spinlock_wait_cycles":25344,"thread_start_us":1043,"threads_started":1}
I20260923 20:09:50.094734 9819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:09:50.079118 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47884 (local address 127.1.170.129:34387)
0923 20:09:50.079934 (+ 816us) server_negotiation.cc:206] Beginning negotiation
0923 20:09:50.079947 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0923 20:09:50.080654 (+ 707us) server_negotiation.cc:407] Connection header received
0923 20:09:50.082308 (+ 1654us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:09:50.082317 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:09:50.082589 (+ 272us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:09:50.082844 (+ 255us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:09:50.084789 (+ 1945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:09:50.087036 (+ 2247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:09:50.091034 (+ 3998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:09:50.092367 (+ 1333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:09:50.092502 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:09:50.092680 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0923 20:09:50.094274 (+ 1594us) server_negotiation.cc:299] Negotiation successful
0923 20:09:50.094463 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":628,"thread_start_us":443,"threads_started":1}
I20260923 20:09:50.097177 9730 tablet_service.cc:1467] Processing CreateTablet for tablet ed83f3f78e0c4a23b339ec475aee56e1 (DEFAULT_TABLE table=test-table [id=9a134f3f862042048d9bd083db7f8558]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 20:09:50.097537 9452 tablet_service.cc:1467] Processing CreateTablet for tablet ed83f3f78e0c4a23b339ec475aee56e1 (DEFAULT_TABLE table=test-table [id=9a134f3f862042048d9bd083db7f8558]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 20:09:50.098768 9730 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ed83f3f78e0c4a23b339ec475aee56e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:09:50.098831 9452 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ed83f3f78e0c4a23b339ec475aee56e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:09:50.099639 9823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:09:50.082519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.130:35035 (local address 127.0.0.1:57368)
0923 20:09:50.090549 (+ 8030us) negotiation.cc:107] Waiting for socket to connect
0923 20:09:50.090571 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0923 20:09:50.090690 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:09:50.091333 (+ 643us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:09:50.091337 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:09:50.091365 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:09:50.091724 (+ 359us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:09:50.091734 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:09:50.093600 (+ 1866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:09:50.093604 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:09:50.096644 (+ 3040us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:09:50.096662 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:09:50.098823 (+ 2161us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:09:50.098830 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:09:50.099002 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:09:50.099034 (+ 32us) client_negotiation.cc:769] Sending connection context
0923 20:09:50.099143 (+ 109us) client_negotiation.cc:240] Negotiation successful
0923 20:09:50.099283 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7888,"thread_start_us":7673,"threads_started":1}
I20260923 20:09:50.099813 9824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:09:50.082630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57368 (local address 127.1.170.130:35035)
0923 20:09:50.090671 (+ 8041us) server_negotiation.cc:206] Beginning negotiation
0923 20:09:50.090682 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0923 20:09:50.090738 (+ 56us) server_negotiation.cc:407] Connection header received
0923 20:09:50.090862 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:09:50.090871 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:09:50.091133 (+ 262us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:09:50.091353 (+ 220us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:09:50.091948 (+ 595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:09:50.093416 (+ 1468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:09:50.096828 (+ 3412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:09:50.098321 (+ 1493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:09:50.098472 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:09:50.098664 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0923 20:09:50.099369 (+ 705us) server_negotiation.cc:299] Negotiation successful
0923 20:09:50.099544 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7884,"thread_start_us":7171,"threads_started":1}
I20260923 20:09:50.103638 9591 tablet_service.cc:1467] Processing CreateTablet for tablet ed83f3f78e0c4a23b339ec475aee56e1 (DEFAULT_TABLE table=test-table [id=9a134f3f862042048d9bd083db7f8558]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 20:09:50.105697 9591 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ed83f3f78e0c4a23b339ec475aee56e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:09:50.110844 9825 tablet_bootstrap.cc:492] T ed83f3f78e0c4a23b339ec475aee56e1 P 3176a36c35d2480fb87fa57226b3bd7d: Bootstrap starting.
I20260923 20:09:50.114598 9826 tablet_bootstrap.cc:492] T ed83f3f78e0c4a23b339ec475aee56e1 P c8c21250d251462898a337ffe611517e: Bootstrap starting.
I20260923 20:09:50.115025 9825 tablet_bootstrap.cc:654] T ed83f3f78e0c4a23b339ec475aee56e1 P 3176a36c35d2480fb87fa57226b3bd7d: Neither blocks nor log segments found. Creating new log.
I20260923 20:09:50.116506 9825 log.cc:824] T ed83f3f78e0c4a23b339ec475aee56e1 P 3176a36c35d2480fb87fa57226b3bd7d: Log is configured to *not* fsync() on all Append() calls
I20260923 20:09:50.118157 9826 tablet_bootstrap.cc:654] T ed83f3f78e0c4a23b339ec475aee56e1 P c8c21250d251462898a337ffe611517e: Neither blocks nor log segments found. Creating new log.
I20260923 20:09:50.119414 9826 log.cc:824] T ed83f3f78e0c4a23b339ec475aee56e1 P c8c21250d251462898a337ffe611517e: Log is configured to *not* fsync() on all Append() calls
I20260923 20:09:50.119777 9825 tablet_bootstrap.cc:492] T ed83f3f78e0c4a23b339ec475aee56e1 P 3176a36c35d2480fb87fa57226b3bd7d: No bootstrap required, opened a new log
I20260923 20:09:50.119902 9828 tablet_bootstrap.cc:492] T ed83f3f78e0c4a23b339ec475aee56e1 P f7f31863969a4bda9e128166c15ac1ad: Bootstrap starting.
I20260923 20:09:50.120565 9825 ts_tablet_manager.cc:1397] T ed83f3f78e0c4a23b339ec475aee56e1 P 3176a36c35d2480fb87fa57226b3bd7d: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260923 20:09:50.122874 9826 tablet_bootstrap.cc:492] T ed83f3f78e0c4a23b339ec475aee56e1 P c8c21250d251462898a337ffe611517e: No bootstrap required, opened a new log
I20260923 20:09:50.123157 9826 ts_tablet_manager.cc:1397] T ed83f3f78e0c4a23b339ec475aee56e1 P c8c21250d251462898a337ffe611517e: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260923 20:09:50.123667 9828 tablet_bootstrap.cc:654] T ed83f3f78e0c4a23b339ec475aee56e1 P f7f31863969a4bda9e128166c15ac1ad: Neither blocks nor log segments found. Creating new log.
I20260923 20:09:50.125056 9828 log.cc:824] T ed83f3f78e0c4a23b339ec475aee56e1 P f7f31863969a4bda9e128166c15ac1ad: Log is configured to *not* fsync() on all Append() calls
I20260923 20:09:50.128422 9828 tablet_bootstrap.cc:492] T ed83f3f78e0c4a23b339ec475aee56e1 P f7f31863969a4bda9e128166c15ac1ad: No bootstrap required, opened a new log
I20260923 20:09:50.128711 9828 ts_tablet_manager.cc:1397] T ed83f3f78e0c4a23b339ec475aee56e1 P f7f31863969a4bda9e128166c15ac1ad: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260923 20:09:50.129237 9825 raft_consensus.cc:348] T ed83f3f78e0c4a23b339ec475aee56e1 P 3176a36c35d2480fb87fa57226b3bd7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8c21250d251462898a337ffe611517e" member_type: VOTER last_known_addr { host: "127.1.170.129" port: 34387 } } peers { permanent_uuid: "3176a36c35d2480fb87fa57226b3bd7d" member_type: VOTER last_known_addr { host: "127.1.170.131" port: 45757 } } peers { permanent_uuid: "f7f31863969a4bda9e128166c15ac1ad" member_type: VOTER last_known_addr { host: "127.1.170.130" port: 35035 } }
I20260923 20:09:50.129621 9825 raft_consensus.cc:374] T ed83f3f78e0c4a23b339ec475aee56e1 P 3176a36c35d2480fb87fa57226b3bd7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:09:50.129751 9825 raft_consensus.cc:729] T ed83f3f78e0c4a23b339ec475aee56e1 P 3176a36c35d2480fb87fa57226b3bd7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3176a36c35d2480fb87fa57226b3bd7d, State: Initialized, Role: FOLLOWER
I20260923 20:09:50.130396 9825 consensus_queue.cc:260] T ed83f3f78e0c4a23b339ec475aee56e1 P 3176a36c35d2480fb87fa57226b3bd7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8c21250d251462898a337ffe611517e" member_type: VOTER last_known_addr { host: "127.1.170.129" port: 34387 } } peers { permanent_uuid: "3176a36c35d2480fb87fa57226b3bd7d" member_type: VOTER last_known_addr { host: "127.1.170.131" port: 45757 } } peers { permanent_uuid: "f7f31863969a4bda9e128166c15ac1ad" member_type: VOTER last_known_addr { host: "127.1.170.130" port: 35035 } }
I20260923 20:09:50.132153 9825 ts_tablet_manager.cc:1428] T ed83f3f78e0c4a23b339ec475aee56e1 P 3176a36c35d2480fb87fa57226b3bd7d: Time spent starting tablet: real 0.011s user 0.010s sys 0.000s
I20260923 20:09:50.133826 9826 raft_consensus.cc:348] T ed83f3f78e0c4a23b339ec475aee56e1 P c8c21250d251462898a337ffe611517e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8c21250d251462898a337ffe611517e" member_type: VOTER last_known_addr { host: "127.1.170.129" port: 34387 } } peers { permanent_uuid: "3176a36c35d2480fb87fa57226b3bd7d" member_type: VOTER last_known_addr { host: "127.1.170.131" port: 45757 } } peers { permanent_uuid: "f7f31863969a4bda9e128166c15ac1ad" member_type: VOTER last_known_addr { host: "127.1.170.130" port: 35035 } }
I20260923 20:09:50.134335 9826 raft_consensus.cc:374] T ed83f3f78e0c4a23b339ec475aee56e1 P c8c21250d251462898a337ffe611517e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:09:50.134438 9826 raft_consensus.cc:729] T ed83f3f78e0c4a23b339ec475aee56e1 P c8c21250d251462898a337ffe611517e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8c21250d251462898a337ffe611517e, State: Initialized, Role: FOLLOWER
I20260923 20:09:50.135067 9826 consensus_queue.cc:260] T ed83f3f78e0c4a23b339ec475aee56e1 P c8c21250d251462898a337ffe611517e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8c21250d251462898a337ffe611517e" member_type: VOTER last_known_addr { host: "127.1.170.129" port: 34387 } } peers { permanent_uuid: "3176a36c35d2480fb87fa57226b3bd7d" member_type: VOTER last_known_addr { host: "127.1.170.131" port: 45757 } } peers { permanent_uuid: "f7f31863969a4bda9e128166c15ac1ad" member_type: VOTER last_known_addr { host: "127.1.170.130" port: 35035 } }
I20260923 20:09:50.137149 9826 ts_tablet_manager.cc:1428] T ed83f3f78e0c4a23b339ec475aee56e1 P c8c21250d251462898a337ffe611517e: Time spent starting tablet: real 0.014s user 0.010s sys 0.004s
I20260923 20:09:50.141572 9828 raft_consensus.cc:348] T ed83f3f78e0c4a23b339ec475aee56e1 P f7f31863969a4bda9e128166c15ac1ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8c21250d251462898a337ffe611517e" member_type: VOTER last_known_addr { host: "127.1.170.129" port: 34387 } } peers { permanent_uuid: "3176a36c35d2480fb87fa57226b3bd7d" member_type: VOTER last_known_addr { host: "127.1.170.131" port: 45757 } } peers { permanent_uuid: "f7f31863969a4bda9e128166c15ac1ad" member_type: VOTER last_known_addr { host: "127.1.170.130" port: 35035 } }
I20260923 20:09:50.142518 9828 raft_consensus.cc:374] T ed83f3f78e0c4a23b339ec475aee56e1 P f7f31863969a4bda9e128166c15ac1ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:09:50.142652 9828 raft_consensus.cc:729] T ed83f3f78e0c4a23b339ec475aee56e1 P f7f31863969a4bda9e128166c15ac1ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f7f31863969a4bda9e128166c15ac1ad, State: Initialized, Role: FOLLOWER
I20260923 20:09:50.143252 9828 consensus_queue.cc:260] T ed83f3f78e0c4a23b339ec475aee56e1 P f7f31863969a4bda9e128166c15ac1ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8c21250d251462898a337ffe611517e" member_type: VOTER last_known_addr { host: "127.1.170.129" port: 34387 } } peers { permanent_uuid: "3176a36c35d2480fb87fa57226b3bd7d" member_type: VOTER last_known_addr { host: "127.1.170.131" port: 45757 } } peers { permanent_uuid: "f7f31863969a4bda9e128166c15ac1ad" member_type: VOTER last_known_addr { host: "127.1.170.130" port: 35035 } }
I20260923 20:09:50.144879 9828 ts_tablet_manager.cc:1428] T ed83f3f78e0c4a23b339ec475aee56e1 P f7f31863969a4bda9e128166c15ac1ad: Time spent starting tablet: real 0.016s user 0.011s sys 0.004s
W20260923 20:09:50.355844 9523 tablet.cc:2369] T ed83f3f78e0c4a23b339ec475aee56e1 P c8c21250d251462898a337ffe611517e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260923 20:09:50.362872 9661 tablet.cc:2369] T ed83f3f78e0c4a23b339ec475aee56e1 P f7f31863969a4bda9e128166c15ac1ad: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 20:09:50.376200 9833 raft_consensus.cc:482] T ed83f3f78e0c4a23b339ec475aee56e1 P f7f31863969a4bda9e128166c15ac1ad [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 20:09:50.376472 9833 raft_consensus.cc:504] T ed83f3f78e0c4a23b339ec475aee56e1 P f7f31863969a4bda9e128166c15ac1ad [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8c21250d251462898a337ffe611517e" member_type: VOTER last_known_addr { host: "127.1.170.129" port: 34387 } } peers { permanent_uuid: "3176a36c35d2480fb87fa57226b3bd7d" member_type: VOTER last_known_addr { host: "127.1.170.131" port: 45757 } } peers { permanent_uuid: "f7f31863969a4bda9e128166c15ac1ad" member_type: VOTER last_known_addr { host: "127.1.170.130" port: 35035 } }
I20260923 20:09:50.378388 9833 leader_election.cc:288] T ed83f3f78e0c4a23b339ec475aee56e1 P f7f31863969a4bda9e128166c15ac1ad [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c8c21250d251462898a337ffe611517e (127.1.170.129:34387), 3176a36c35d2480fb87fa57226b3bd7d (127.1.170.131:45757)
W20260923 20:09:50.381174 9801 tablet.cc:2369] T ed83f3f78e0c4a23b339ec475aee56e1 P 3176a36c35d2480fb87fa57226b3bd7d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 20:09:50.387552 9819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:09:50.378778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.170.130:46133 (local address 127.1.170.129:34387)
0923 20:09:50.378983 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0923 20:09:50.378990 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:09:50.380023 (+ 1033us) server_negotiation.cc:407] Connection header received
0923 20:09:50.380121 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:09:50.380126 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:09:50.380193 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:09:50.380321 (+ 128us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:09:50.381159 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:09:50.382037 (+ 878us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:09:50.385021 (+ 2984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:09:50.386279 (+ 1258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:09:50.386470 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:09:50.386545 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0923 20:09:50.387153 (+ 608us) server_negotiation.cc:299] Negotiation successful
0923 20:09:50.387306 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260923 20:09:50.387573 9822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:09:50.379250 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.170.130:33491 (local address 127.1.170.131:45757)
0923 20:09:50.379456 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0923 20:09:50.379461 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 20:09:50.379832 (+ 371us) server_negotiation.cc:407] Connection header received
0923 20:09:50.380088 (+ 256us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:09:50.380093 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:09:50.380177 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:09:50.380285 (+ 108us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:09:50.381110 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:09:50.382187 (+ 1077us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:09:50.385021 (+ 2834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:09:50.386279 (+ 1258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:09:50.386470 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:09:50.386545 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0923 20:09:50.387247 (+ 702us) server_negotiation.cc:299] Negotiation successful
0923 20:09:50.387368 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260923 20:09:50.387812 9835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:09:50.379028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.129:34387 (local address 127.1.170.130:46133)
0923 20:09:50.379900 (+ 872us) negotiation.cc:107] Waiting for socket to connect
0923 20:09:50.379915 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0923 20:09:50.380027 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:09:50.380348 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:09:50.380354 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:09:50.380385 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:09:50.380938 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:09:50.380945 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:09:50.382215 (+ 1270us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:09:50.382219 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:09:50.384823 (+ 2604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:09:50.384839 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:09:50.386662 (+ 1823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:09:50.386668 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:09:50.386938 (+ 270us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:09:50.386985 (+ 47us) client_negotiation.cc:769] Sending connection context
0923 20:09:50.387110 (+ 125us) client_negotiation.cc:240] Negotiation successful
0923 20:09:50.387303 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":771,"mutex_wait_us":203,"thread_start_us":591,"threads_started":1}
I20260923 20:09:50.388329 9834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:09:50.378820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.131:45757 (local address 127.1.170.130:33491)
0923 20:09:50.379687 (+ 867us) negotiation.cc:107] Waiting for socket to connect
0923 20:09:50.379709 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0923 20:09:50.379871 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:09:50.380348 (+ 477us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:09:50.380358 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:09:50.380452 (+ 94us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:09:50.380931 (+ 479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:09:50.380945 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:09:50.382305 (+ 1360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:09:50.382312 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:09:50.384823 (+ 2511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:09:50.384839 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:09:50.386828 (+ 1989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:09:50.386837 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:09:50.387035 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:09:50.387092 (+ 57us) client_negotiation.cc:769] Sending connection context
0923 20:09:50.387740 (+ 648us) client_negotiation.cc:240] Negotiation successful
0923 20:09:50.387919 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":672,"mutex_wait_us":133,"spinlock_wait_cycles":25472,"thread_start_us":451,"threads_started":1}
I20260923 20:09:50.389062 9477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed83f3f78e0c4a23b339ec475aee56e1" candidate_uuid: "f7f31863969a4bda9e128166c15ac1ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8c21250d251462898a337ffe611517e" is_pre_election: true
I20260923 20:09:50.389703 9477 raft_consensus.cc:2454] T ed83f3f78e0c4a23b339ec475aee56e1 P c8c21250d251462898a337ffe611517e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f7f31863969a4bda9e128166c15ac1ad in term 0.
I20260923 20:09:50.389675 9755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed83f3f78e0c4a23b339ec475aee56e1" candidate_uuid: "f7f31863969a4bda9e128166c15ac1ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3176a36c35d2480fb87fa57226b3bd7d" is_pre_election: true
I20260923 20:09:50.390254 9755 raft_consensus.cc:2454] T ed83f3f78e0c4a23b339ec475aee56e1 P 3176a36c35d2480fb87fa57226b3bd7d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f7f31863969a4bda9e128166c15ac1ad in term 0.
I20260923 20:09:50.390445 9542 leader_election.cc:302] T ed83f3f78e0c4a23b339ec475aee56e1 P f7f31863969a4bda9e128166c15ac1ad [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c8c21250d251462898a337ffe611517e, f7f31863969a4bda9e128166c15ac1ad; no voters:
I20260923 20:09:50.390940 9833 raft_consensus.cc:2781] T ed83f3f78e0c4a23b339ec475aee56e1 P f7f31863969a4bda9e128166c15ac1ad [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 20:09:50.391100 9833 raft_consensus.cc:482] T ed83f3f78e0c4a23b339ec475aee56e1 P f7f31863969a4bda9e128166c15ac1ad [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 20:09:50.391233 9833 raft_consensus.cc:3037] T ed83f3f78e0c4a23b339ec475aee56e1 P f7f31863969a4bda9e128166c15ac1ad [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:09:50.393559 9833 raft_consensus.cc:504] T ed83f3f78e0c4a23b339ec475aee56e1 P f7f31863969a4bda9e128166c15ac1ad [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8c21250d251462898a337ffe611517e" member_type: VOTER last_known_addr { host: "127.1.170.129" port: 34387 } } peers { permanent_uuid: "3176a36c35d2480fb87fa57226b3bd7d" member_type: VOTER last_known_addr { host: "127.1.170.131" port: 45757 } } peers { permanent_uuid: "f7f31863969a4bda9e128166c15ac1ad" member_type: VOTER last_known_addr { host: "127.1.170.130" port: 35035 } }
I20260923 20:09:50.394323 9833 leader_election.cc:288] T ed83f3f78e0c4a23b339ec475aee56e1 P f7f31863969a4bda9e128166c15ac1ad [CANDIDATE]: Term 1 election: Requested vote from peers c8c21250d251462898a337ffe611517e (127.1.170.129:34387), 3176a36c35d2480fb87fa57226b3bd7d (127.1.170.131:45757)
I20260923 20:09:50.394878 9477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed83f3f78e0c4a23b339ec475aee56e1" candidate_uuid: "f7f31863969a4bda9e128166c15ac1ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8c21250d251462898a337ffe611517e"
I20260923 20:09:50.394912 9755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed83f3f78e0c4a23b339ec475aee56e1" candidate_uuid: "f7f31863969a4bda9e128166c15ac1ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3176a36c35d2480fb87fa57226b3bd7d"
I20260923 20:09:50.395108 9755 raft_consensus.cc:3037] T ed83f3f78e0c4a23b339ec475aee56e1 P 3176a36c35d2480fb87fa57226b3bd7d [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:09:50.395109 9477 raft_consensus.cc:3037] T ed83f3f78e0c4a23b339ec475aee56e1 P c8c21250d251462898a337ffe611517e [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:09:50.398449 9755 raft_consensus.cc:2454] T ed83f3f78e0c4a23b339ec475aee56e1 P 3176a36c35d2480fb87fa57226b3bd7d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f7f31863969a4bda9e128166c15ac1ad in term 1.
I20260923 20:09:50.398449 9477 raft_consensus.cc:2454] T ed83f3f78e0c4a23b339ec475aee56e1 P c8c21250d251462898a337ffe611517e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f7f31863969a4bda9e128166c15ac1ad in term 1.
I20260923 20:09:50.399092 9544 leader_election.cc:302] T ed83f3f78e0c4a23b339ec475aee56e1 P f7f31863969a4bda9e128166c15ac1ad [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3176a36c35d2480fb87fa57226b3bd7d, f7f31863969a4bda9e128166c15ac1ad; no voters:
I20260923 20:09:50.399482 9833 raft_consensus.cc:2781] T ed83f3f78e0c4a23b339ec475aee56e1 P f7f31863969a4bda9e128166c15ac1ad [term 1 FOLLOWER]: Leader election won for term 1
I20260923 20:09:50.399969 9833 raft_consensus.cc:686] T ed83f3f78e0c4a23b339ec475aee56e1 P f7f31863969a4bda9e128166c15ac1ad [term 1 LEADER]: Becoming Leader. State: Replica: f7f31863969a4bda9e128166c15ac1ad, State: Running, Role: LEADER
I20260923 20:09:50.400458 9833 consensus_queue.cc:237] T ed83f3f78e0c4a23b339ec475aee56e1 P f7f31863969a4bda9e128166c15ac1ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8c21250d251462898a337ffe611517e" member_type: VOTER last_known_addr { host: "127.1.170.129" port: 34387 } } peers { permanent_uuid: "3176a36c35d2480fb87fa57226b3bd7d" member_type: VOTER last_known_addr { host: "127.1.170.131" port: 45757 } } peers { permanent_uuid: "f7f31863969a4bda9e128166c15ac1ad" member_type: VOTER last_known_addr { host: "127.1.170.130" port: 35035 } }
I20260923 20:09:50.405040 9314 catalog_manager.cc:5495] T ed83f3f78e0c4a23b339ec475aee56e1 P f7f31863969a4bda9e128166c15ac1ad reported cstate change: term changed from 0 to 1, leader changed from <none> to f7f31863969a4bda9e128166c15ac1ad (127.1.170.130). New cstate: current_term: 1 leader_uuid: "f7f31863969a4bda9e128166c15ac1ad" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8c21250d251462898a337ffe611517e" member_type: VOTER last_known_addr { host: "127.1.170.129" port: 34387 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3176a36c35d2480fb87fa57226b3bd7d" member_type: VOTER last_known_addr { host: "127.1.170.131" port: 45757 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f7f31863969a4bda9e128166c15ac1ad" member_type: VOTER last_known_addr { host: "127.1.170.130" port: 35035 } health_report { overall_health: HEALTHY } } }
I20260923 20:09:50.464124 9824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:09:50.458023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57374 (local address 127.1.170.130:35035)
0923 20:09:50.458286 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0923 20:09:50.458291 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 20:09:50.458309 (+ 18us) server_negotiation.cc:407] Connection header received
0923 20:09:50.458618 (+ 309us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:09:50.458624 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:09:50.458738 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:09:50.458929 (+ 191us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 20:09:50.459308 (+ 379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:09:50.460116 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:09:50.461531 (+ 1415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:09:50.462050 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:09:50.462164 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:09:50.462672 (+ 508us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 20:09:50.463326 (+ 654us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 20:09:50.463448 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0923 20:09:50.463738 (+ 290us) server_negotiation.cc:299] Negotiation successful
0923 20:09:50.463818 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260923 20:09:50.475564 9755 raft_consensus.cc:1261] T ed83f3f78e0c4a23b339ec475aee56e1 P 3176a36c35d2480fb87fa57226b3bd7d [term 1 FOLLOWER]: Refusing update from remote peer f7f31863969a4bda9e128166c15ac1ad: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 20:09:50.475569 9477 raft_consensus.cc:1261] T ed83f3f78e0c4a23b339ec475aee56e1 P c8c21250d251462898a337ffe611517e [term 1 FOLLOWER]: Refusing update from remote peer f7f31863969a4bda9e128166c15ac1ad: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 20:09:50.476665 9836 consensus_queue.cc:1035] T ed83f3f78e0c4a23b339ec475aee56e1 P f7f31863969a4bda9e128166c15ac1ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "3176a36c35d2480fb87fa57226b3bd7d" member_type: VOTER last_known_addr { host: "127.1.170.131" port: 45757 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260923 20:09:50.477351 9833 consensus_queue.cc:1035] T ed83f3f78e0c4a23b339ec475aee56e1 P f7f31863969a4bda9e128166c15ac1ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8c21250d251462898a337ffe611517e" member_type: VOTER last_known_addr { host: "127.1.170.129" port: 34387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 20:09:50.491741 9839 mvcc.cc:205] Tried to move back new op lower bound from 7332635404175929344 to 7332635403884265472. Current Snapshot: MvccSnapshot[applied={T|T < 7332635404175929344}]
I20260923 20:09:50.506815 9314 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50408:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 20:09:50.507143 9314 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50408:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 20:09:50.511271 9314 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6b22557909954a558ccd24b8fbbc4f78: Sending DeleteTablet for 3 replicas of tablet ed83f3f78e0c4a23b339ec475aee56e1
I20260923 20:09:50.512809 9452 tablet_service.cc:1514] Processing DeleteTablet for tablet ed83f3f78e0c4a23b339ec475aee56e1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 20:09:50 UTC) from {username='oryx', principal='oryx/127.1.170.190@KRBTEST.COM'} at 127.0.0.1:47884
I20260923 20:09:50.513367 9730 tablet_service.cc:1514] Processing DeleteTablet for tablet ed83f3f78e0c4a23b339ec475aee56e1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 20:09:50 UTC) from {username='oryx', principal='oryx/127.1.170.190@KRBTEST.COM'} at 127.0.0.1:59158
I20260923 20:09:50.513705 9591 tablet_service.cc:1514] Processing DeleteTablet for tablet ed83f3f78e0c4a23b339ec475aee56e1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 20:09:50 UTC) from {username='oryx', principal='oryx/127.1.170.190@KRBTEST.COM'} at 127.0.0.1:57368
I20260923 20:09:50.518785 9314 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50408:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 20:09:50.519580 9314 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 20:09:50.522439 9849 tablet_replica.cc:330] T ed83f3f78e0c4a23b339ec475aee56e1 P c8c21250d251462898a337ffe611517e: stopping tablet replica
I20260923 20:09:50.522948 9849 raft_consensus.cc:2229] T ed83f3f78e0c4a23b339ec475aee56e1 P c8c21250d251462898a337ffe611517e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 20:09:50.523514 9849 raft_consensus.cc:2258] T ed83f3f78e0c4a23b339ec475aee56e1 P c8c21250d251462898a337ffe611517e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 20:09:50.523645 9851 tablet_replica.cc:330] T ed83f3f78e0c4a23b339ec475aee56e1 P f7f31863969a4bda9e128166c15ac1ad: stopping tablet replica
I20260923 20:09:50.524012 9850 tablet_replica.cc:330] T ed83f3f78e0c4a23b339ec475aee56e1 P 3176a36c35d2480fb87fa57226b3bd7d: stopping tablet replica
I20260923 20:09:50.524044 9851 raft_consensus.cc:2229] T ed83f3f78e0c4a23b339ec475aee56e1 P f7f31863969a4bda9e128166c15ac1ad [term 1 LEADER]: Raft consensus shutting down.
I20260923 20:09:50.524435 9850 raft_consensus.cc:2229] T ed83f3f78e0c4a23b339ec475aee56e1 P 3176a36c35d2480fb87fa57226b3bd7d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 20:09:50.524770 9851 raft_consensus.cc:2258] T ed83f3f78e0c4a23b339ec475aee56e1 P f7f31863969a4bda9e128166c15ac1ad [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 20:09:50.524978 9850 raft_consensus.cc:2258] T ed83f3f78e0c4a23b339ec475aee56e1 P 3176a36c35d2480fb87fa57226b3bd7d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 20:09:50.526901 9850 ts_tablet_manager.cc:1905] T ed83f3f78e0c4a23b339ec475aee56e1 P 3176a36c35d2480fb87fa57226b3bd7d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 20:09:50.529502 9851 ts_tablet_manager.cc:1905] T ed83f3f78e0c4a23b339ec475aee56e1 P f7f31863969a4bda9e128166c15ac1ad: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 20:09:50.530956 9452 tablet_service.cc:1467] Processing CreateTablet for tablet 292b3ffc765a4e348e71cf99035dbef3 (DEFAULT_TABLE table=test-table [id=4a527c130a1744edba000c0bb44df839]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 20:09:50.530956 9591 tablet_service.cc:1467] Processing CreateTablet for tablet 292b3ffc765a4e348e71cf99035dbef3 (DEFAULT_TABLE table=test-table [id=4a527c130a1744edba000c0bb44df839]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 20:09:50.531404 9452 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 292b3ffc765a4e348e71cf99035dbef3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:09:50.531421 9591 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 292b3ffc765a4e348e71cf99035dbef3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:09:50.535729 9729 tablet_service.cc:1467] Processing CreateTablet for tablet 292b3ffc765a4e348e71cf99035dbef3 (DEFAULT_TABLE table=test-table [id=4a527c130a1744edba000c0bb44df839]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 20:09:50.536306 9729 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 292b3ffc765a4e348e71cf99035dbef3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:09:50.538933 9850 ts_tablet_manager.cc:1918] T ed83f3f78e0c4a23b339ec475aee56e1 P 3176a36c35d2480fb87fa57226b3bd7d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 20:09:50.539181 9850 log.cc:1196] T ed83f3f78e0c4a23b339ec475aee56e1 P 3176a36c35d2480fb87fa57226b3bd7d: Deleting WAL directory at /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790194121577027-1706-0/minicluster-data/ts-2/wal/wals/ed83f3f78e0c4a23b339ec475aee56e1
I20260923 20:09:50.539529 9828 tablet_bootstrap.cc:492] T 292b3ffc765a4e348e71cf99035dbef3 P f7f31863969a4bda9e128166c15ac1ad: Bootstrap starting.
I20260923 20:09:50.540053 9850 ts_tablet_manager.cc:1939] T ed83f3f78e0c4a23b339ec475aee56e1 P 3176a36c35d2480fb87fa57226b3bd7d: Deleting consensus metadata
I20260923 20:09:50.540133 9849 ts_tablet_manager.cc:1905] T ed83f3f78e0c4a23b339ec475aee56e1 P c8c21250d251462898a337ffe611517e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 20:09:50.541301 9828 tablet_bootstrap.cc:654] T 292b3ffc765a4e348e71cf99035dbef3 P f7f31863969a4bda9e128166c15ac1ad: Neither blocks nor log segments found. Creating new log.
I20260923 20:09:50.542410 9301 catalog_manager.cc:4841] TS 3176a36c35d2480fb87fa57226b3bd7d (127.1.170.131:45757): tablet ed83f3f78e0c4a23b339ec475aee56e1 (table test-table [id=9a134f3f862042048d9bd083db7f8558]) successfully deleted
I20260923 20:09:50.543963 9828 tablet_bootstrap.cc:492] T 292b3ffc765a4e348e71cf99035dbef3 P f7f31863969a4bda9e128166c15ac1ad: No bootstrap required, opened a new log
I20260923 20:09:50.544123 9828 ts_tablet_manager.cc:1397] T 292b3ffc765a4e348e71cf99035dbef3 P f7f31863969a4bda9e128166c15ac1ad: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20260923 20:09:50.545089 9828 raft_consensus.cc:348] T 292b3ffc765a4e348e71cf99035dbef3 P f7f31863969a4bda9e128166c15ac1ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8c21250d251462898a337ffe611517e" member_type: VOTER last_known_addr { host: "127.1.170.129" port: 34387 } } peers { permanent_uuid: "f7f31863969a4bda9e128166c15ac1ad" member_type: VOTER last_known_addr { host: "127.1.170.130" port: 35035 } } peers { permanent_uuid: "3176a36c35d2480fb87fa57226b3bd7d" member_type: VOTER last_known_addr { host: "127.1.170.131" port: 45757 } }
I20260923 20:09:50.545313 9828 raft_consensus.cc:374] T 292b3ffc765a4e348e71cf99035dbef3 P f7f31863969a4bda9e128166c15ac1ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:09:50.545631 9828 raft_consensus.cc:729] T 292b3ffc765a4e348e71cf99035dbef3 P f7f31863969a4bda9e128166c15ac1ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f7f31863969a4bda9e128166c15ac1ad, State: Initialized, Role: FOLLOWER
I20260923 20:09:50.546355 9826 tablet_bootstrap.cc:492] T 292b3ffc765a4e348e71cf99035dbef3 P c8c21250d251462898a337ffe611517e: Bootstrap starting.
I20260923 20:09:50.547302 9825 tablet_bootstrap.cc:492] T 292b3ffc765a4e348e71cf99035dbef3 P 3176a36c35d2480fb87fa57226b3bd7d: Bootstrap starting.
I20260923 20:09:50.547443 9849 ts_tablet_manager.cc:1918] T ed83f3f78e0c4a23b339ec475aee56e1 P c8c21250d251462898a337ffe611517e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 20:09:50.547622 9849 log.cc:1196] T ed83f3f78e0c4a23b339ec475aee56e1 P c8c21250d251462898a337ffe611517e: Deleting WAL directory at /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790194121577027-1706-0/minicluster-data/ts-0/wal/wals/ed83f3f78e0c4a23b339ec475aee56e1
I20260923 20:09:50.548038 9826 tablet_bootstrap.cc:654] T 292b3ffc765a4e348e71cf99035dbef3 P c8c21250d251462898a337ffe611517e: Neither blocks nor log segments found. Creating new log.
I20260923 20:09:50.545969 9828 consensus_queue.cc:260] T 292b3ffc765a4e348e71cf99035dbef3 P f7f31863969a4bda9e128166c15ac1ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8c21250d251462898a337ffe611517e" member_type: VOTER last_known_addr { host: "127.1.170.129" port: 34387 } } peers { permanent_uuid: "f7f31863969a4bda9e128166c15ac1ad" member_type: VOTER last_known_addr { host: "127.1.170.130" port: 35035 } } peers { permanent_uuid: "3176a36c35d2480fb87fa57226b3bd7d" member_type: VOTER last_known_addr { host: "127.1.170.131" port: 45757 } }
I20260923 20:09:50.548231 9849 ts_tablet_manager.cc:1939] T ed83f3f78e0c4a23b339ec475aee56e1 P c8c21250d251462898a337ffe611517e: Deleting consensus metadata
I20260923 20:09:50.548803 9828 ts_tablet_manager.cc:1428] T 292b3ffc765a4e348e71cf99035dbef3 P f7f31863969a4bda9e128166c15ac1ad: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20260923 20:09:50.548974 9825 tablet_bootstrap.cc:654] T 292b3ffc765a4e348e71cf99035dbef3 P 3176a36c35d2480fb87fa57226b3bd7d: Neither blocks nor log segments found. Creating new log.
I20260923 20:09:50.550146 9299 catalog_manager.cc:4841] TS c8c21250d251462898a337ffe611517e (127.1.170.129:34387): tablet ed83f3f78e0c4a23b339ec475aee56e1 (table test-table [id=9a134f3f862042048d9bd083db7f8558]) successfully deleted
I20260923 20:09:50.555301 9825 tablet_bootstrap.cc:492] T 292b3ffc765a4e348e71cf99035dbef3 P 3176a36c35d2480fb87fa57226b3bd7d: No bootstrap required, opened a new log
I20260923 20:09:50.555430 9826 tablet_bootstrap.cc:492] T 292b3ffc765a4e348e71cf99035dbef3 P c8c21250d251462898a337ffe611517e: No bootstrap required, opened a new log
I20260923 20:09:50.555475 9825 ts_tablet_manager.cc:1397] T 292b3ffc765a4e348e71cf99035dbef3 P 3176a36c35d2480fb87fa57226b3bd7d: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.004s
I20260923 20:09:50.555585 9826 ts_tablet_manager.cc:1397] T 292b3ffc765a4e348e71cf99035dbef3 P c8c21250d251462898a337ffe611517e: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.003s
I20260923 20:09:50.556313 9825 raft_consensus.cc:348] T 292b3ffc765a4e348e71cf99035dbef3 P 3176a36c35d2480fb87fa57226b3bd7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8c21250d251462898a337ffe611517e" member_type: VOTER last_known_addr { host: "127.1.170.129" port: 34387 } } peers { permanent_uuid: "f7f31863969a4bda9e128166c15ac1ad" member_type: VOTER last_known_addr { host: "127.1.170.130" port: 35035 } } peers { permanent_uuid: "3176a36c35d2480fb87fa57226b3bd7d" member_type: VOTER last_known_addr { host: "127.1.170.131" port: 45757 } }
I20260923 20:09:50.556385 9826 raft_consensus.cc:348] T 292b3ffc765a4e348e71cf99035dbef3 P c8c21250d251462898a337ffe611517e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8c21250d251462898a337ffe611517e" member_type: VOTER last_known_addr { host: "127.1.170.129" port: 34387 } } peers { permanent_uuid: "f7f31863969a4bda9e128166c15ac1ad" member_type: VOTER last_known_addr { host: "127.1.170.130" port: 35035 } } peers { permanent_uuid: "3176a36c35d2480fb87fa57226b3bd7d" member_type: VOTER last_known_addr { host: "127.1.170.131" port: 45757 } }
I20260923 20:09:50.556557 9825 raft_consensus.cc:374] T 292b3ffc765a4e348e71cf99035dbef3 P 3176a36c35d2480fb87fa57226b3bd7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:09:50.556598 9826 raft_consensus.cc:374] T 292b3ffc765a4e348e71cf99035dbef3 P c8c21250d251462898a337ffe611517e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:09:50.556643 9825 raft_consensus.cc:729] T 292b3ffc765a4e348e71cf99035dbef3 P 3176a36c35d2480fb87fa57226b3bd7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3176a36c35d2480fb87fa57226b3bd7d, State: Initialized, Role: FOLLOWER
I20260923 20:09:50.556703 9826 raft_consensus.cc:729] T 292b3ffc765a4e348e71cf99035dbef3 P c8c21250d251462898a337ffe611517e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8c21250d251462898a337ffe611517e, State: Initialized, Role: FOLLOWER
I20260923 20:09:50.556896 9825 consensus_queue.cc:260] T 292b3ffc765a4e348e71cf99035dbef3 P 3176a36c35d2480fb87fa57226b3bd7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8c21250d251462898a337ffe611517e" member_type: VOTER last_known_addr { host: "127.1.170.129" port: 34387 } } peers { permanent_uuid: "f7f31863969a4bda9e128166c15ac1ad" member_type: VOTER last_known_addr { host: "127.1.170.130" port: 35035 } } peers { permanent_uuid: "3176a36c35d2480fb87fa57226b3bd7d" member_type: VOTER last_known_addr { host: "127.1.170.131" port: 45757 } }
I20260923 20:09:50.557089 9826 consensus_queue.cc:260] T 292b3ffc765a4e348e71cf99035dbef3 P c8c21250d251462898a337ffe611517e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8c21250d251462898a337ffe611517e" member_type: VOTER last_known_addr { host: "127.1.170.129" port: 34387 } } peers { permanent_uuid: "f7f31863969a4bda9e128166c15ac1ad" member_type: VOTER last_known_addr { host: "127.1.170.130" port: 35035 } } peers { permanent_uuid: "3176a36c35d2480fb87fa57226b3bd7d" member_type: VOTER last_known_addr { host: "127.1.170.131" port: 45757 } }
I20260923 20:09:50.557444 9825 ts_tablet_manager.cc:1428] T 292b3ffc765a4e348e71cf99035dbef3 P 3176a36c35d2480fb87fa57226b3bd7d: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260923 20:09:50.558467 9851 ts_tablet_manager.cc:1918] T ed83f3f78e0c4a23b339ec475aee56e1 P f7f31863969a4bda9e128166c15ac1ad: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 20:09:50.558653 9851 log.cc:1196] T ed83f3f78e0c4a23b339ec475aee56e1 P f7f31863969a4bda9e128166c15ac1ad: Deleting WAL directory at /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790194121577027-1706-0/minicluster-data/ts-1/wal/wals/ed83f3f78e0c4a23b339ec475aee56e1
I20260923 20:09:50.559281 9851 ts_tablet_manager.cc:1939] T ed83f3f78e0c4a23b339ec475aee56e1 P f7f31863969a4bda9e128166c15ac1ad: Deleting consensus metadata
I20260923 20:09:50.560446 9826 ts_tablet_manager.cc:1428] T 292b3ffc765a4e348e71cf99035dbef3 P c8c21250d251462898a337ffe611517e: Time spent starting tablet: real 0.005s user 0.000s sys 0.002s
I20260923 20:09:50.560906 9299 catalog_manager.cc:4841] TS f7f31863969a4bda9e128166c15ac1ad (127.1.170.130:35035): tablet ed83f3f78e0c4a23b339ec475aee56e1 (table test-table [id=9a134f3f862042048d9bd083db7f8558]) successfully deleted
I20260923 20:09:50.759001 9831 raft_consensus.cc:482] T 292b3ffc765a4e348e71cf99035dbef3 P 3176a36c35d2480fb87fa57226b3bd7d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 20:09:50.759341 9831 raft_consensus.cc:504] T 292b3ffc765a4e348e71cf99035dbef3 P 3176a36c35d2480fb87fa57226b3bd7d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8c21250d251462898a337ffe611517e" member_type: VOTER last_known_addr { host: "127.1.170.129" port: 34387 } } peers { permanent_uuid: "f7f31863969a4bda9e128166c15ac1ad" member_type: VOTER last_known_addr { host: "127.1.170.130" port: 35035 } } peers { permanent_uuid: "3176a36c35d2480fb87fa57226b3bd7d" member_type: VOTER last_known_addr { host: "127.1.170.131" port: 45757 } }
I20260923 20:09:50.760887 9831 leader_election.cc:288] T 292b3ffc765a4e348e71cf99035dbef3 P 3176a36c35d2480fb87fa57226b3bd7d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c8c21250d251462898a337ffe611517e (127.1.170.129:34387), f7f31863969a4bda9e128166c15ac1ad (127.1.170.130:35035)
I20260923 20:09:50.768429 9819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:09:50.761420 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.170.131:57009 (local address 127.1.170.129:34387)
0923 20:09:50.761629 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0923 20:09:50.761637 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 20:09:50.762046 (+ 409us) server_negotiation.cc:407] Connection header received
0923 20:09:50.762317 (+ 271us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:09:50.762322 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:09:50.762402 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:09:50.762470 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:09:50.763379 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:09:50.764413 (+ 1034us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:09:50.766403 (+ 1990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:09:50.767235 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:09:50.767444 (+ 209us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:09:50.767524 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0923 20:09:50.767958 (+ 434us) server_negotiation.cc:299] Negotiation successful
0923 20:09:50.768101 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260923 20:09:50.768821 9855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:09:50.761314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.129:34387 (local address 127.1.170.131:57009)
0923 20:09:50.761929 (+ 615us) negotiation.cc:107] Waiting for socket to connect
0923 20:09:50.761954 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0923 20:09:50.762136 (+ 182us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:09:50.762715 (+ 579us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:09:50.762722 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:09:50.762761 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:09:50.763173 (+ 412us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:09:50.763184 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:09:50.764550 (+ 1366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:09:50.764556 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:09:50.766243 (+ 1687us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:09:50.766254 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:09:50.767475 (+ 1221us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:09:50.767481 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:09:50.767766 (+ 285us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:09:50.767793 (+ 27us) client_negotiation.cc:769] Sending connection context
0923 20:09:50.767935 (+ 142us) client_negotiation.cc:240] Negotiation successful
0923 20:09:50.768102 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":465,"mutex_wait_us":88,"thread_start_us":319,"threads_started":1}
I20260923 20:09:50.769656 9477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "292b3ffc765a4e348e71cf99035dbef3" candidate_uuid: "3176a36c35d2480fb87fa57226b3bd7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8c21250d251462898a337ffe611517e" is_pre_election: true
I20260923 20:09:50.769963 9477 raft_consensus.cc:2454] T 292b3ffc765a4e348e71cf99035dbef3 P c8c21250d251462898a337ffe611517e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3176a36c35d2480fb87fa57226b3bd7d in term 0.
I20260923 20:09:50.770231 9824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:09:50.762311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.170.131:41923 (local address 127.1.170.130:35035)
0923 20:09:50.762513 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0923 20:09:50.762520 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:09:50.763178 (+ 658us) server_negotiation.cc:407] Connection header received
0923 20:09:50.763368 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:09:50.763374 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:09:50.763494 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:09:50.763601 (+ 107us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:09:50.764203 (+ 602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:09:50.765362 (+ 1159us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:09:50.768021 (+ 2659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:09:50.769324 (+ 1303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:09:50.769483 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:09:50.769551 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0923 20:09:50.769818 (+ 267us) server_negotiation.cc:299] Negotiation successful
0923 20:09:50.769971 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260923 20:09:50.770275 9856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:09:50.762107 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.130:35035 (local address 127.1.170.131:41923)
0923 20:09:50.763041 (+ 934us) negotiation.cc:107] Waiting for socket to connect
0923 20:09:50.763066 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0923 20:09:50.763214 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:09:50.763674 (+ 460us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:09:50.763679 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:09:50.763707 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:09:50.764046 (+ 339us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:09:50.764057 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:09:50.765543 (+ 1486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:09:50.765549 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:09:50.767815 (+ 2266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:09:50.767831 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:09:50.769534 (+ 1703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:09:50.769539 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:09:50.769662 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:09:50.769685 (+ 23us) client_negotiation.cc:769] Sending connection context
0923 20:09:50.769760 (+ 75us) client_negotiation.cc:240] Negotiation successful
0923 20:09:50.769909 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":783,"mutex_wait_us":57,"thread_start_us":557,"threads_started":1}
I20260923 20:09:50.770596 9681 leader_election.cc:302] T 292b3ffc765a4e348e71cf99035dbef3 P 3176a36c35d2480fb87fa57226b3bd7d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3176a36c35d2480fb87fa57226b3bd7d, c8c21250d251462898a337ffe611517e; no voters:
I20260923 20:09:50.770972 9831 raft_consensus.cc:2781] T 292b3ffc765a4e348e71cf99035dbef3 P 3176a36c35d2480fb87fa57226b3bd7d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 20:09:50.771082 9831 raft_consensus.cc:482] T 292b3ffc765a4e348e71cf99035dbef3 P 3176a36c35d2480fb87fa57226b3bd7d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 20:09:50.771173 9831 raft_consensus.cc:3037] T 292b3ffc765a4e348e71cf99035dbef3 P 3176a36c35d2480fb87fa57226b3bd7d [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:09:50.771552 9611 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "292b3ffc765a4e348e71cf99035dbef3" candidate_uuid: "3176a36c35d2480fb87fa57226b3bd7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f7f31863969a4bda9e128166c15ac1ad" is_pre_election: true
I20260923 20:09:50.771919 9611 raft_consensus.cc:2454] T 292b3ffc765a4e348e71cf99035dbef3 P f7f31863969a4bda9e128166c15ac1ad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3176a36c35d2480fb87fa57226b3bd7d in term 0.
I20260923 20:09:50.774590 9831 raft_consensus.cc:504] T 292b3ffc765a4e348e71cf99035dbef3 P 3176a36c35d2480fb87fa57226b3bd7d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8c21250d251462898a337ffe611517e" member_type: VOTER last_known_addr { host: "127.1.170.129" port: 34387 } } peers { permanent_uuid: "f7f31863969a4bda9e128166c15ac1ad" member_type: VOTER last_known_addr { host: "127.1.170.130" port: 35035 } } peers { permanent_uuid: "3176a36c35d2480fb87fa57226b3bd7d" member_type: VOTER last_known_addr { host: "127.1.170.131" port: 45757 } }
I20260923 20:09:50.775341 9831 leader_election.cc:288] T 292b3ffc765a4e348e71cf99035dbef3 P 3176a36c35d2480fb87fa57226b3bd7d [CANDIDATE]: Term 1 election: Requested vote from peers c8c21250d251462898a337ffe611517e (127.1.170.129:34387), f7f31863969a4bda9e128166c15ac1ad (127.1.170.130:35035)
I20260923 20:09:50.776005 9477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "292b3ffc765a4e348e71cf99035dbef3" candidate_uuid: "3176a36c35d2480fb87fa57226b3bd7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8c21250d251462898a337ffe611517e"
I20260923 20:09:50.776083 9611 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "292b3ffc765a4e348e71cf99035dbef3" candidate_uuid: "3176a36c35d2480fb87fa57226b3bd7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f7f31863969a4bda9e128166c15ac1ad"
I20260923 20:09:50.776247 9477 raft_consensus.cc:3037] T 292b3ffc765a4e348e71cf99035dbef3 P c8c21250d251462898a337ffe611517e [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:09:50.776311 9611 raft_consensus.cc:3037] T 292b3ffc765a4e348e71cf99035dbef3 P f7f31863969a4bda9e128166c15ac1ad [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:09:50.779410 9477 raft_consensus.cc:2454] T 292b3ffc765a4e348e71cf99035dbef3 P c8c21250d251462898a337ffe611517e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3176a36c35d2480fb87fa57226b3bd7d in term 1.
I20260923 20:09:50.779913 9681 leader_election.cc:302] T 292b3ffc765a4e348e71cf99035dbef3 P 3176a36c35d2480fb87fa57226b3bd7d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3176a36c35d2480fb87fa57226b3bd7d, c8c21250d251462898a337ffe611517e; no voters:
I20260923 20:09:50.780269 9831 raft_consensus.cc:2781] T 292b3ffc765a4e348e71cf99035dbef3 P 3176a36c35d2480fb87fa57226b3bd7d [term 1 FOLLOWER]: Leader election won for term 1
I20260923 20:09:50.780328 9611 raft_consensus.cc:2454] T 292b3ffc765a4e348e71cf99035dbef3 P f7f31863969a4bda9e128166c15ac1ad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3176a36c35d2480fb87fa57226b3bd7d in term 1.
I20260923 20:09:50.780786 9831 raft_consensus.cc:686] T 292b3ffc765a4e348e71cf99035dbef3 P 3176a36c35d2480fb87fa57226b3bd7d [term 1 LEADER]: Becoming Leader. State: Replica: 3176a36c35d2480fb87fa57226b3bd7d, State: Running, Role: LEADER
I20260923 20:09:50.781231 9831 consensus_queue.cc:237] T 292b3ffc765a4e348e71cf99035dbef3 P 3176a36c35d2480fb87fa57226b3bd7d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8c21250d251462898a337ffe611517e" member_type: VOTER last_known_addr { host: "127.1.170.129" port: 34387 } } peers { permanent_uuid: "f7f31863969a4bda9e128166c15ac1ad" member_type: VOTER last_known_addr { host: "127.1.170.130" port: 35035 } } peers { permanent_uuid: "3176a36c35d2480fb87fa57226b3bd7d" member_type: VOTER last_known_addr { host: "127.1.170.131" port: 45757 } }
I20260923 20:09:50.785537 9314 catalog_manager.cc:5495] T 292b3ffc765a4e348e71cf99035dbef3 P 3176a36c35d2480fb87fa57226b3bd7d reported cstate change: term changed from 0 to 1, leader changed from <none> to 3176a36c35d2480fb87fa57226b3bd7d (127.1.170.131). New cstate: current_term: 1 leader_uuid: "3176a36c35d2480fb87fa57226b3bd7d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8c21250d251462898a337ffe611517e" member_type: VOTER last_known_addr { host: "127.1.170.129" port: 34387 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f7f31863969a4bda9e128166c15ac1ad" member_type: VOTER last_known_addr { host: "127.1.170.130" port: 35035 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3176a36c35d2480fb87fa57226b3bd7d" member_type: VOTER last_known_addr { host: "127.1.170.131" port: 45757 } health_report { overall_health: HEALTHY } } }
Sep 23 20:09:50 dist-test-slave-th4g krb5kdc[9270](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194187, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.170.190@KRBTEST.COM for oryx/127.1.170.190@KRBTEST.COM
I20260923 20:09:50.886454 9865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:09:50.864884 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.190:45507 (local address 127.0.0.1:50410)
0923 20:09:50.871431 (+ 6547us) negotiation.cc:107] Waiting for socket to connect
0923 20:09:50.871464 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0923 20:09:50.871791 (+ 327us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:09:50.872249 (+ 458us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:09:50.872255 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:09:50.873090 (+ 835us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:09:50.873330 (+ 240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:09:50.873340 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:09:50.874524 (+ 1184us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:09:50.874532 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:09:50.875633 (+ 1101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:09:50.875647 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:09:50.876458 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:09:50.876464 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:09:50.876664 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:09:50.877404 (+ 740us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:09:50.877421 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:09:50.879681 (+ 2260us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:09:50.883112 (+ 3431us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:09:50.883129 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:09:50.883138 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:09:50.883583 (+ 445us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:09:50.883973 (+ 390us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:09:50.883978 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:09:50.883982 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:09:50.884131 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:09:50.884684 (+ 553us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:09:50.884693 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:09:50.884859 (+ 166us) client_negotiation.cc:769] Sending connection context
0923 20:09:50.885244 (+ 385us) client_negotiation.cc:240] Negotiation successful
0923 20:09:50.885784 (+ 540us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6346,"thread_start_us":6164,"threads_started":1}
I20260923 20:09:50.886493 9864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:09:50.864906 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50410 (local address 127.1.170.190:45507)
0923 20:09:50.871978 (+ 7072us) server_negotiation.cc:206] Beginning negotiation
0923 20:09:50.871982 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 20:09:50.871999 (+ 17us) server_negotiation.cc:407] Connection header received
0923 20:09:50.872040 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:09:50.872043 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:09:50.872097 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:09:50.872219 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:09:50.873529 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:09:50.874390 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:09:50.875819 (+ 1429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:09:50.876247 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:09:50.876408 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:09:50.879904 (+ 3496us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:09:50.879909 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:09:50.879912 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:09:50.879939 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:09:50.882956 (+ 3017us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:09:50.883710 (+ 754us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:09:50.883718 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:09:50.883721 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:09:50.883859 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:09:50.884253 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:09:50.884261 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:09:50.884265 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:09:50.884539 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:09:50.884715 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0923 20:09:50.885394 (+ 679us) server_negotiation.cc:299] Negotiation successful
0923 20:09:50.886042 (+ 648us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":200,"server-negotiator.queue_time_us":6934,"thread_start_us":6714,"threads_started":1}
I20260923 20:09:50.893455 9314 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.1.170.190@KRBTEST.COM'} at 127.0.0.1:50410:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260923 20:09:50.895112 9314 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 20:09:50.906410 9591 tablet_service.cc:1467] Processing CreateTablet for tablet bc11985d54a54be0a8f3f94e55db422d (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=925345cd4d394921a3bb46be14135792]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260923 20:09:50.906404 9452 tablet_service.cc:1467] Processing CreateTablet for tablet bc11985d54a54be0a8f3f94e55db422d (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=925345cd4d394921a3bb46be14135792]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260923 20:09:50.907172 9591 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bc11985d54a54be0a8f3f94e55db422d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:09:50.907194 9452 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bc11985d54a54be0a8f3f94e55db422d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:09:50.911913 9729 tablet_service.cc:1467] Processing CreateTablet for tablet bc11985d54a54be0a8f3f94e55db422d (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=925345cd4d394921a3bb46be14135792]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260923 20:09:50.912590 9729 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bc11985d54a54be0a8f3f94e55db422d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:09:50.916154 9828 tablet_bootstrap.cc:492] T bc11985d54a54be0a8f3f94e55db422d P f7f31863969a4bda9e128166c15ac1ad: Bootstrap starting.
I20260923 20:09:50.918059 9828 tablet_bootstrap.cc:654] T bc11985d54a54be0a8f3f94e55db422d P f7f31863969a4bda9e128166c15ac1ad: Neither blocks nor log segments found. Creating new log.
I20260923 20:09:50.918346 9826 tablet_bootstrap.cc:492] T bc11985d54a54be0a8f3f94e55db422d P c8c21250d251462898a337ffe611517e: Bootstrap starting.
I20260923 20:09:50.920156 9826 tablet_bootstrap.cc:654] T bc11985d54a54be0a8f3f94e55db422d P c8c21250d251462898a337ffe611517e: Neither blocks nor log segments found. Creating new log.
I20260923 20:09:50.920423 9828 tablet_bootstrap.cc:492] T bc11985d54a54be0a8f3f94e55db422d P f7f31863969a4bda9e128166c15ac1ad: No bootstrap required, opened a new log
I20260923 20:09:50.920593 9828 ts_tablet_manager.cc:1397] T bc11985d54a54be0a8f3f94e55db422d P f7f31863969a4bda9e128166c15ac1ad: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20260923 20:09:50.920729 9825 tablet_bootstrap.cc:492] T bc11985d54a54be0a8f3f94e55db422d P 3176a36c35d2480fb87fa57226b3bd7d: Bootstrap starting.
I20260923 20:09:50.921464 9828 raft_consensus.cc:348] T bc11985d54a54be0a8f3f94e55db422d P f7f31863969a4bda9e128166c15ac1ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8c21250d251462898a337ffe611517e" member_type: VOTER last_known_addr { host: "127.1.170.129" port: 34387 } } peers { permanent_uuid: "f7f31863969a4bda9e128166c15ac1ad" member_type: VOTER last_known_addr { host: "127.1.170.130" port: 35035 } } peers { permanent_uuid: "3176a36c35d2480fb87fa57226b3bd7d" member_type: VOTER last_known_addr { host: "127.1.170.131" port: 45757 } }
I20260923 20:09:50.921828 9828 raft_consensus.cc:374] T bc11985d54a54be0a8f3f94e55db422d P f7f31863969a4bda9e128166c15ac1ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:09:50.921948 9828 raft_consensus.cc:729] T bc11985d54a54be0a8f3f94e55db422d P f7f31863969a4bda9e128166c15ac1ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f7f31863969a4bda9e128166c15ac1ad, State: Initialized, Role: FOLLOWER
I20260923 20:09:50.922328 9825 tablet_bootstrap.cc:654] T bc11985d54a54be0a8f3f94e55db422d P 3176a36c35d2480fb87fa57226b3bd7d: Neither blocks nor log segments found. Creating new log.
I20260923 20:09:50.922262 9828 consensus_queue.cc:260] T bc11985d54a54be0a8f3f94e55db422d P f7f31863969a4bda9e128166c15ac1ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8c21250d251462898a337ffe611517e" member_type: VOTER last_known_addr { host: "127.1.170.129" port: 34387 } } peers { permanent_uuid: "f7f31863969a4bda9e128166c15ac1ad" member_type: VOTER last_known_addr { host: "127.1.170.130" port: 35035 } } peers { permanent_uuid: "3176a36c35d2480fb87fa57226b3bd7d" member_type: VOTER last_known_addr { host: "127.1.170.131" port: 45757 } }
I20260923 20:09:50.922847 9844 tablet_replica.cc:439] T bc11985d54a54be0a8f3f94e55db422d P f7f31863969a4bda9e128166c15ac1ad: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8c21250d251462898a337ffe611517e" member_type: VOTER last_known_addr { host: "127.1.170.129" port: 34387 } } peers { permanent_uuid: "f7f31863969a4bda9e128166c15ac1ad" member_type: VOTER last_known_addr { host: "127.1.170.130" port: 35035 } } peers { permanent_uuid: "3176a36c35d2480fb87fa57226b3bd7d" member_type: VOTER last_known_addr { host: "127.1.170.131" port: 45757 } } }
I20260923 20:09:50.923259 9844 tablet_replica.cc:442] T bc11985d54a54be0a8f3f94e55db422d P f7f31863969a4bda9e128166c15ac1ad: This TxnStatusTablet replica's current role is: FOLLOWER
I20260923 20:09:50.923867 9828 ts_tablet_manager.cc:1428] T bc11985d54a54be0a8f3f94e55db422d P f7f31863969a4bda9e128166c15ac1ad: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260923 20:09:50.927906 9826 tablet_bootstrap.cc:492] T bc11985d54a54be0a8f3f94e55db422d P c8c21250d251462898a337ffe611517e: No bootstrap required, opened a new log
I20260923 20:09:50.928083 9826 ts_tablet_manager.cc:1397] T bc11985d54a54be0a8f3f94e55db422d P c8c21250d251462898a337ffe611517e: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.004s
I20260923 20:09:50.928859 9826 raft_consensus.cc:348] T bc11985d54a54be0a8f3f94e55db422d P c8c21250d251462898a337ffe611517e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8c21250d251462898a337ffe611517e" member_type: VOTER last_known_addr { host: "127.1.170.129" port: 34387 } } peers { permanent_uuid: "f7f31863969a4bda9e128166c15ac1ad" member_type: VOTER last_known_addr { host: "127.1.170.130" port: 35035 } } peers { permanent_uuid: "3176a36c35d2480fb87fa57226b3bd7d" member_type: VOTER last_known_addr { host: "127.1.170.131" port: 45757 } }
I20260923 20:09:50.929104 9826 raft_consensus.cc:374] T bc11985d54a54be0a8f3f94e55db422d P c8c21250d251462898a337ffe611517e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:09:50.929181 9826 raft_consensus.cc:729] T bc11985d54a54be0a8f3f94e55db422d P c8c21250d251462898a337ffe611517e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8c21250d251462898a337ffe611517e, State: Initialized, Role: FOLLOWER
I20260923 20:09:50.929446 9826 consensus_queue.cc:260] T bc11985d54a54be0a8f3f94e55db422d P c8c21250d251462898a337ffe611517e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8c21250d251462898a337ffe611517e" member_type: VOTER last_known_addr { host: "127.1.170.129" port: 34387 } } peers { permanent_uuid: "f7f31863969a4bda9e128166c15ac1ad" member_type: VOTER last_known_addr { host: "127.1.170.130" port: 35035 } } peers { permanent_uuid: "3176a36c35d2480fb87fa57226b3bd7d" member_type: VOTER last_known_addr { host: "127.1.170.131" port: 45757 } }
I20260923 20:09:50.929942 9832 tablet_replica.cc:439] T bc11985d54a54be0a8f3f94e55db422d P c8c21250d251462898a337ffe611517e: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8c21250d251462898a337ffe611517e" member_type: VOTER last_known_addr { host: "127.1.170.129" port: 34387 } } peers { permanent_uuid: "f7f31863969a4bda9e128166c15ac1ad" member_type: VOTER last_known_addr { host: "127.1.170.130" port: 35035 } } peers { permanent_uuid: "3176a36c35d2480fb87fa57226b3bd7d" member_type: VOTER last_known_addr { host: "127.1.170.131" port: 45757 } } }
I20260923 20:09:50.930295 9832 tablet_replica.cc:442] T bc11985d54a54be0a8f3f94e55db422d P c8c21250d251462898a337ffe611517e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260923 20:09:50.932816 9826 ts_tablet_manager.cc:1428] T bc11985d54a54be0a8f3f94e55db422d P c8c21250d251462898a337ffe611517e: Time spent starting tablet: real 0.005s user 0.002s sys 0.001s
I20260923 20:09:50.935590 9825 tablet_bootstrap.cc:492] T bc11985d54a54be0a8f3f94e55db422d P 3176a36c35d2480fb87fa57226b3bd7d: No bootstrap required, opened a new log
I20260923 20:09:50.935797 9825 ts_tablet_manager.cc:1397] T bc11985d54a54be0a8f3f94e55db422d P 3176a36c35d2480fb87fa57226b3bd7d: Time spent bootstrapping tablet: real 0.015s user 0.003s sys 0.009s
I20260923 20:09:50.936620 9825 raft_consensus.cc:348] T bc11985d54a54be0a8f3f94e55db422d P 3176a36c35d2480fb87fa57226b3bd7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8c21250d251462898a337ffe611517e" member_type: VOTER last_known_addr { host: "127.1.170.129" port: 34387 } } peers { permanent_uuid: "f7f31863969a4bda9e128166c15ac1ad" member_type: VOTER last_known_addr { host: "127.1.170.130" port: 35035 } } peers { permanent_uuid: "3176a36c35d2480fb87fa57226b3bd7d" member_type: VOTER last_known_addr { host: "127.1.170.131" port: 45757 } }
I20260923 20:09:50.936887 9825 raft_consensus.cc:374] T bc11985d54a54be0a8f3f94e55db422d P 3176a36c35d2480fb87fa57226b3bd7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:09:50.936978 9825 raft_consensus.cc:729] T bc11985d54a54be0a8f3f94e55db422d P 3176a36c35d2480fb87fa57226b3bd7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3176a36c35d2480fb87fa57226b3bd7d, State: Initialized, Role: FOLLOWER
I20260923 20:09:50.937244 9825 consensus_queue.cc:260] T bc11985d54a54be0a8f3f94e55db422d P 3176a36c35d2480fb87fa57226b3bd7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8c21250d251462898a337ffe611517e" member_type: VOTER last_known_addr { host: "127.1.170.129" port: 34387 } } peers { permanent_uuid: "f7f31863969a4bda9e128166c15ac1ad" member_type: VOTER last_known_addr { host: "127.1.170.130" port: 35035 } } peers { permanent_uuid: "3176a36c35d2480fb87fa57226b3bd7d" member_type: VOTER last_known_addr { host: "127.1.170.131" port: 45757 } }
I20260923 20:09:50.938020 9825 ts_tablet_manager.cc:1428] T bc11985d54a54be0a8f3f94e55db422d P 3176a36c35d2480fb87fa57226b3bd7d: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260923 20:09:50.937968 9831 tablet_replica.cc:439] T bc11985d54a54be0a8f3f94e55db422d P 3176a36c35d2480fb87fa57226b3bd7d: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8c21250d251462898a337ffe611517e" member_type: VOTER last_known_addr { host: "127.1.170.129" port: 34387 } } peers { permanent_uuid: "f7f31863969a4bda9e128166c15ac1ad" member_type: VOTER last_known_addr { host: "127.1.170.130" port: 35035 } } peers { permanent_uuid: "3176a36c35d2480fb87fa57226b3bd7d" member_type: VOTER last_known_addr { host: "127.1.170.131" port: 45757 } } }
I20260923 20:09:50.938280 9831 tablet_replica.cc:442] T bc11985d54a54be0a8f3f94e55db422d P 3176a36c35d2480fb87fa57226b3bd7d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260923 20:09:50.940191 9844 raft_consensus.cc:482] T bc11985d54a54be0a8f3f94e55db422d P f7f31863969a4bda9e128166c15ac1ad [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 20:09:50.940418 9844 raft_consensus.cc:504] T bc11985d54a54be0a8f3f94e55db422d P f7f31863969a4bda9e128166c15ac1ad [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8c21250d251462898a337ffe611517e" member_type: VOTER last_known_addr { host: "127.1.170.129" port: 34387 } } peers { permanent_uuid: "f7f31863969a4bda9e128166c15ac1ad" member_type: VOTER last_known_addr { host: "127.1.170.130" port: 35035 } } peers { permanent_uuid: "3176a36c35d2480fb87fa57226b3bd7d" member_type: VOTER last_known_addr { host: "127.1.170.131" port: 45757 } }
I20260923 20:09:50.941435 9844 leader_election.cc:288] T bc11985d54a54be0a8f3f94e55db422d P f7f31863969a4bda9e128166c15ac1ad [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c8c21250d251462898a337ffe611517e (127.1.170.129:34387), 3176a36c35d2480fb87fa57226b3bd7d (127.1.170.131:45757)
I20260923 20:09:50.941844 9477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc11985d54a54be0a8f3f94e55db422d" candidate_uuid: "f7f31863969a4bda9e128166c15ac1ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8c21250d251462898a337ffe611517e" is_pre_election: true
I20260923 20:09:50.942195 9477 raft_consensus.cc:2454] T bc11985d54a54be0a8f3f94e55db422d P c8c21250d251462898a337ffe611517e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f7f31863969a4bda9e128166c15ac1ad in term 0.
I20260923 20:09:50.942371 9755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc11985d54a54be0a8f3f94e55db422d" candidate_uuid: "f7f31863969a4bda9e128166c15ac1ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3176a36c35d2480fb87fa57226b3bd7d" is_pre_election: true
I20260923 20:09:50.942608 9755 raft_consensus.cc:2454] T bc11985d54a54be0a8f3f94e55db422d P 3176a36c35d2480fb87fa57226b3bd7d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f7f31863969a4bda9e128166c15ac1ad in term 0.
I20260923 20:09:50.942736 9542 leader_election.cc:302] T bc11985d54a54be0a8f3f94e55db422d P f7f31863969a4bda9e128166c15ac1ad [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c8c21250d251462898a337ffe611517e, f7f31863969a4bda9e128166c15ac1ad; no voters:
I20260923 20:09:50.943049 9844 raft_consensus.cc:2781] T bc11985d54a54be0a8f3f94e55db422d P f7f31863969a4bda9e128166c15ac1ad [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 20:09:50.943194 9844 raft_consensus.cc:482] T bc11985d54a54be0a8f3f94e55db422d P f7f31863969a4bda9e128166c15ac1ad [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 20:09:50.943305 9844 raft_consensus.cc:3037] T bc11985d54a54be0a8f3f94e55db422d P f7f31863969a4bda9e128166c15ac1ad [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:09:50.946534 9844 raft_consensus.cc:504] T bc11985d54a54be0a8f3f94e55db422d P f7f31863969a4bda9e128166c15ac1ad [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8c21250d251462898a337ffe611517e" member_type: VOTER last_known_addr { host: "127.1.170.129" port: 34387 } } peers { permanent_uuid: "f7f31863969a4bda9e128166c15ac1ad" member_type: VOTER last_known_addr { host: "127.1.170.130" port: 35035 } } peers { permanent_uuid: "3176a36c35d2480fb87fa57226b3bd7d" member_type: VOTER last_known_addr { host: "127.1.170.131" port: 45757 } }
I20260923 20:09:50.947181 9844 leader_election.cc:288] T bc11985d54a54be0a8f3f94e55db422d P f7f31863969a4bda9e128166c15ac1ad [CANDIDATE]: Term 1 election: Requested vote from peers c8c21250d251462898a337ffe611517e (127.1.170.129:34387), 3176a36c35d2480fb87fa57226b3bd7d (127.1.170.131:45757)
I20260923 20:09:50.947737 9755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc11985d54a54be0a8f3f94e55db422d" candidate_uuid: "f7f31863969a4bda9e128166c15ac1ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3176a36c35d2480fb87fa57226b3bd7d"
I20260923 20:09:50.947989 9755 raft_consensus.cc:3037] T bc11985d54a54be0a8f3f94e55db422d P 3176a36c35d2480fb87fa57226b3bd7d [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:09:50.947995 9477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc11985d54a54be0a8f3f94e55db422d" candidate_uuid: "f7f31863969a4bda9e128166c15ac1ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8c21250d251462898a337ffe611517e"
I20260923 20:09:50.948223 9477 raft_consensus.cc:3037] T bc11985d54a54be0a8f3f94e55db422d P c8c21250d251462898a337ffe611517e [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:09:50.951161 9477 raft_consensus.cc:2454] T bc11985d54a54be0a8f3f94e55db422d P c8c21250d251462898a337ffe611517e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f7f31863969a4bda9e128166c15ac1ad in term 1.
I20260923 20:09:50.951174 9755 raft_consensus.cc:2454] T bc11985d54a54be0a8f3f94e55db422d P 3176a36c35d2480fb87fa57226b3bd7d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f7f31863969a4bda9e128166c15ac1ad in term 1.
I20260923 20:09:50.951649 9544 leader_election.cc:302] T bc11985d54a54be0a8f3f94e55db422d P f7f31863969a4bda9e128166c15ac1ad [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 3176a36c35d2480fb87fa57226b3bd7d, c8c21250d251462898a337ffe611517e, f7f31863969a4bda9e128166c15ac1ad; no voters:
I20260923 20:09:50.952054 9844 raft_consensus.cc:2781] T bc11985d54a54be0a8f3f94e55db422d P f7f31863969a4bda9e128166c15ac1ad [term 1 FOLLOWER]: Leader election won for term 1
I20260923 20:09:50.952248 9844 raft_consensus.cc:686] T bc11985d54a54be0a8f3f94e55db422d P f7f31863969a4bda9e128166c15ac1ad [term 1 LEADER]: Becoming Leader. State: Replica: f7f31863969a4bda9e128166c15ac1ad, State: Running, Role: LEADER
I20260923 20:09:50.952571 9844 consensus_queue.cc:237] T bc11985d54a54be0a8f3f94e55db422d P f7f31863969a4bda9e128166c15ac1ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8c21250d251462898a337ffe611517e" member_type: VOTER last_known_addr { host: "127.1.170.129" port: 34387 } } peers { permanent_uuid: "f7f31863969a4bda9e128166c15ac1ad" member_type: VOTER last_known_addr { host: "127.1.170.130" port: 35035 } } peers { permanent_uuid: "3176a36c35d2480fb87fa57226b3bd7d" member_type: VOTER last_known_addr { host: "127.1.170.131" port: 45757 } }
I20260923 20:09:50.954502 9833 tablet_replica.cc:439] T bc11985d54a54be0a8f3f94e55db422d P f7f31863969a4bda9e128166c15ac1ad: TxnStatusTablet state changed. Reason: New leader f7f31863969a4bda9e128166c15ac1ad. Latest consensus state: current_term: 1 leader_uuid: "f7f31863969a4bda9e128166c15ac1ad" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8c21250d251462898a337ffe611517e" member_type: VOTER last_known_addr { host: "127.1.170.129" port: 34387 } } peers { permanent_uuid: "f7f31863969a4bda9e128166c15ac1ad" member_type: VOTER last_known_addr { host: "127.1.170.130" port: 35035 } } peers { permanent_uuid: "3176a36c35d2480fb87fa57226b3bd7d" member_type: VOTER last_known_addr { host: "127.1.170.131" port: 45757 } } }
I20260923 20:09:50.954818 9833 tablet_replica.cc:442] T bc11985d54a54be0a8f3f94e55db422d P f7f31863969a4bda9e128166c15ac1ad: This TxnStatusTablet replica's current role is: LEADER
I20260923 20:09:50.955585 9870 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260923 20:09:50.955825 9870 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260923 20:09:50.956100 9315 catalog_manager.cc:5495] T bc11985d54a54be0a8f3f94e55db422d P f7f31863969a4bda9e128166c15ac1ad reported cstate change: term changed from 0 to 1, leader changed from <none> to f7f31863969a4bda9e128166c15ac1ad (127.1.170.130). New cstate: current_term: 1 leader_uuid: "f7f31863969a4bda9e128166c15ac1ad" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8c21250d251462898a337ffe611517e" member_type: VOTER last_known_addr { host: "127.1.170.129" port: 34387 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f7f31863969a4bda9e128166c15ac1ad" member_type: VOTER last_known_addr { host: "127.1.170.130" port: 35035 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3176a36c35d2480fb87fa57226b3bd7d" member_type: VOTER last_known_addr { host: "127.1.170.131" port: 45757 } health_report { overall_health: UNKNOWN } } }
I20260923 20:09:50.983016 9824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:09:50.977006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57388 (local address 127.1.170.130:35035)
0923 20:09:50.977222 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0923 20:09:50.977227 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 20:09:50.977379 (+ 152us) server_negotiation.cc:407] Connection header received
0923 20:09:50.977692 (+ 313us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:09:50.977696 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:09:50.977764 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:09:50.977850 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 20:09:50.978307 (+ 457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:09:50.979118 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:09:50.981020 (+ 1902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:09:50.981488 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:09:50.981709 (+ 221us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:09:50.982315 (+ 606us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 20:09:50.982430 (+ 115us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 20:09:50.982510 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0923 20:09:50.982709 (+ 199us) server_negotiation.cc:299] Negotiation successful
0923 20:09:50.982781 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260923 20:09:50.983119 9865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:09:50.977001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.130:35035 (local address 127.0.0.1:57388)
0923 20:09:50.977275 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0923 20:09:50.977298 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0923 20:09:50.977471 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:09:50.977933 (+ 462us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:09:50.977938 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:09:50.977952 (+ 14us) client_negotiation.cc:189] Negotiated authn=TOKEN
0923 20:09:50.978128 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:09:50.978138 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:09:50.979255 (+ 1117us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:09:50.979261 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:09:50.980851 (+ 1590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:09:50.980867 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:09:50.981752 (+ 885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:09:50.981759 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:09:50.981991 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:09:50.982071 (+ 80us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0923 20:09:50.982534 (+ 463us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0923 20:09:50.982552 (+ 18us) client_negotiation.cc:769] Sending connection context
0923 20:09:50.982635 (+ 83us) client_negotiation.cc:240] Negotiation successful
0923 20:09:50.982803 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":67}
I20260923 20:09:50.987454 9477 raft_consensus.cc:1261] T bc11985d54a54be0a8f3f94e55db422d P c8c21250d251462898a337ffe611517e [term 1 FOLLOWER]: Refusing update from remote peer f7f31863969a4bda9e128166c15ac1ad: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 20:09:50.987461 9755 raft_consensus.cc:1261] T bc11985d54a54be0a8f3f94e55db422d P 3176a36c35d2480fb87fa57226b3bd7d [term 1 FOLLOWER]: Refusing update from remote peer f7f31863969a4bda9e128166c15ac1ad: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 20:09:50.988114 9844 consensus_queue.cc:1035] T bc11985d54a54be0a8f3f94e55db422d P f7f31863969a4bda9e128166c15ac1ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "3176a36c35d2480fb87fa57226b3bd7d" member_type: VOTER last_known_addr { host: "127.1.170.131" port: 45757 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 20:09:50.988448 9833 consensus_queue.cc:1035] T bc11985d54a54be0a8f3f94e55db422d P f7f31863969a4bda9e128166c15ac1ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8c21250d251462898a337ffe611517e" member_type: VOTER last_known_addr { host: "127.1.170.129" port: 34387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 20:09:50.992089 9832 tablet_replica.cc:439] T bc11985d54a54be0a8f3f94e55db422d P c8c21250d251462898a337ffe611517e: TxnStatusTablet state changed. Reason: New leader f7f31863969a4bda9e128166c15ac1ad. Latest consensus state: current_term: 1 leader_uuid: "f7f31863969a4bda9e128166c15ac1ad" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8c21250d251462898a337ffe611517e" member_type: VOTER last_known_addr { host: "127.1.170.129" port: 34387 } } peers { permanent_uuid: "f7f31863969a4bda9e128166c15ac1ad" member_type: VOTER last_known_addr { host: "127.1.170.130" port: 35035 } } peers { permanent_uuid: "3176a36c35d2480fb87fa57226b3bd7d" member_type: VOTER last_known_addr { host: "127.1.170.131" port: 45757 } } }
I20260923 20:09:50.992489 9832 tablet_replica.cc:442] T bc11985d54a54be0a8f3f94e55db422d P c8c21250d251462898a337ffe611517e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260923 20:09:50.994473 9833 tablet_replica.cc:439] T bc11985d54a54be0a8f3f94e55db422d P f7f31863969a4bda9e128166c15ac1ad: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f7f31863969a4bda9e128166c15ac1ad" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8c21250d251462898a337ffe611517e" member_type: VOTER last_known_addr { host: "127.1.170.129" port: 34387 } } peers { permanent_uuid: "f7f31863969a4bda9e128166c15ac1ad" member_type: VOTER last_known_addr { host: "127.1.170.130" port: 35035 } } peers { permanent_uuid: "3176a36c35d2480fb87fa57226b3bd7d" member_type: VOTER last_known_addr { host: "127.1.170.131" port: 45757 } } }
I20260923 20:09:50.994837 9833 tablet_replica.cc:442] T bc11985d54a54be0a8f3f94e55db422d P f7f31863969a4bda9e128166c15ac1ad: This TxnStatusTablet replica's current role is: LEADER
I20260923 20:09:50.997474 9832 tablet_replica.cc:439] T bc11985d54a54be0a8f3f94e55db422d P c8c21250d251462898a337ffe611517e: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f7f31863969a4bda9e128166c15ac1ad" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8c21250d251462898a337ffe611517e" member_type: VOTER last_known_addr { host: "127.1.170.129" port: 34387 } } peers { permanent_uuid: "f7f31863969a4bda9e128166c15ac1ad" member_type: VOTER last_known_addr { host: "127.1.170.130" port: 35035 } } peers { permanent_uuid: "3176a36c35d2480fb87fa57226b3bd7d" member_type: VOTER last_known_addr { host: "127.1.170.131" port: 45757 } } }
I20260923 20:09:50.997997 9832 tablet_replica.cc:442] T bc11985d54a54be0a8f3f94e55db422d P c8c21250d251462898a337ffe611517e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260923 20:09:50.998095 9831 tablet_replica.cc:439] T bc11985d54a54be0a8f3f94e55db422d P 3176a36c35d2480fb87fa57226b3bd7d: TxnStatusTablet state changed. Reason: New leader f7f31863969a4bda9e128166c15ac1ad. Latest consensus state: current_term: 1 leader_uuid: "f7f31863969a4bda9e128166c15ac1ad" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8c21250d251462898a337ffe611517e" member_type: VOTER last_known_addr { host: "127.1.170.129" port: 34387 } } peers { permanent_uuid: "f7f31863969a4bda9e128166c15ac1ad" member_type: VOTER last_known_addr { host: "127.1.170.130" port: 35035 } } peers { permanent_uuid: "3176a36c35d2480fb87fa57226b3bd7d" member_type: VOTER last_known_addr { host: "127.1.170.131" port: 45757 } } }
I20260923 20:09:50.998704 9831 tablet_replica.cc:442] T bc11985d54a54be0a8f3f94e55db422d P 3176a36c35d2480fb87fa57226b3bd7d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260923 20:09:51.003165 9844 tablet_replica.cc:439] T bc11985d54a54be0a8f3f94e55db422d P f7f31863969a4bda9e128166c15ac1ad: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f7f31863969a4bda9e128166c15ac1ad" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8c21250d251462898a337ffe611517e" member_type: VOTER last_known_addr { host: "127.1.170.129" port: 34387 } } peers { permanent_uuid: "f7f31863969a4bda9e128166c15ac1ad" member_type: VOTER last_known_addr { host: "127.1.170.130" port: 35035 } } peers { permanent_uuid: "3176a36c35d2480fb87fa57226b3bd7d" member_type: VOTER last_known_addr { host: "127.1.170.131" port: 45757 } } }
I20260923 20:09:51.003486 9844 tablet_replica.cc:442] T bc11985d54a54be0a8f3f94e55db422d P f7f31863969a4bda9e128166c15ac1ad: This TxnStatusTablet replica's current role is: LEADER
I20260923 20:09:51.009840 9831 tablet_replica.cc:439] T bc11985d54a54be0a8f3f94e55db422d P 3176a36c35d2480fb87fa57226b3bd7d: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f7f31863969a4bda9e128166c15ac1ad" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8c21250d251462898a337ffe611517e" member_type: VOTER last_known_addr { host: "127.1.170.129" port: 34387 } } peers { permanent_uuid: "f7f31863969a4bda9e128166c15ac1ad" member_type: VOTER last_known_addr { host: "127.1.170.130" port: 35035 } } peers { permanent_uuid: "3176a36c35d2480fb87fa57226b3bd7d" member_type: VOTER last_known_addr { host: "127.1.170.131" port: 45757 } } }
I20260923 20:09:51.010222 9831 tablet_replica.cc:442] T bc11985d54a54be0a8f3f94e55db422d P 3176a36c35d2480fb87fa57226b3bd7d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260923 20:09:51.017834 9876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:09:51.005552 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59160 (local address 127.1.170.131:45757)
0923 20:09:51.006851 (+ 1299us) server_negotiation.cc:206] Beginning negotiation
0923 20:09:51.006864 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0923 20:09:51.009241 (+ 2377us) server_negotiation.cc:407] Connection header received
0923 20:09:51.009481 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:09:51.009486 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:09:51.010835 (+ 1349us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:09:51.011039 (+ 204us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 20:09:51.011922 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:09:51.013144 (+ 1222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:09:51.014911 (+ 1767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:09:51.015591 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:09:51.015731 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:09:51.016283 (+ 552us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 20:09:51.017060 (+ 777us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 20:09:51.017202 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0923 20:09:51.017391 (+ 189us) server_negotiation.cc:299] Negotiation successful
0923 20:09:51.017481 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":397,"thread_start_us":236,"threads_started":1}
I20260923 20:09:51.038471 9824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:09:51.031233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.170.131:44749 (local address 127.1.170.130:35035)
0923 20:09:51.031405 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0923 20:09:51.031410 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 20:09:51.032376 (+ 966us) server_negotiation.cc:407] Connection header received
0923 20:09:51.032644 (+ 268us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:09:51.032650 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:09:51.032730 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:09:51.032841 (+ 111us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 20:09:51.033492 (+ 651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:09:51.034560 (+ 1068us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:09:51.036049 (+ 1489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:09:51.036510 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:09:51.036655 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:09:51.037236 (+ 581us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 20:09:51.037442 (+ 206us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 20:09:51.037544 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0923 20:09:51.037726 (+ 182us) server_negotiation.cc:299] Negotiation successful
0923 20:09:51.037819 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260923 20:09:51.038547 9880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:09:51.031210 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.130:35035 (local address 127.1.170.131:44749)
0923 20:09:51.032202 (+ 992us) negotiation.cc:107] Waiting for socket to connect
0923 20:09:51.032224 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0923 20:09:51.032470 (+ 246us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:09:51.032858 (+ 388us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:09:51.032863 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:09:51.032876 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0923 20:09:51.033275 (+ 399us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:09:51.033285 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:09:51.034704 (+ 1419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:09:51.034710 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:09:51.035899 (+ 1189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:09:51.035911 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:09:51.036730 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:09:51.036739 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:09:51.036952 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:09:51.037029 (+ 77us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0923 20:09:51.037547 (+ 518us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0923 20:09:51.037562 (+ 15us) client_negotiation.cc:769] Sending connection context
0923 20:09:51.037692 (+ 130us) client_negotiation.cc:240] Negotiation successful
0923 20:09:51.037858 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":770,"thread_start_us":551,"threads_started":1}
I20260923 20:09:51.048780 9477 raft_consensus.cc:1261] T 292b3ffc765a4e348e71cf99035dbef3 P c8c21250d251462898a337ffe611517e [term 1 FOLLOWER]: Refusing update from remote peer 3176a36c35d2480fb87fa57226b3bd7d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 20:09:51.048995 9611 raft_consensus.cc:1261] T 292b3ffc765a4e348e71cf99035dbef3 P f7f31863969a4bda9e128166c15ac1ad [term 1 FOLLOWER]: Refusing update from remote peer 3176a36c35d2480fb87fa57226b3bd7d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 20:09:51.049515 9857 consensus_queue.cc:1035] T 292b3ffc765a4e348e71cf99035dbef3 P 3176a36c35d2480fb87fa57226b3bd7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8c21250d251462898a337ffe611517e" member_type: VOTER last_known_addr { host: "127.1.170.129" port: 34387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 20:09:51.049943 9831 consensus_queue.cc:1035] T 292b3ffc765a4e348e71cf99035dbef3 P 3176a36c35d2480fb87fa57226b3bd7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f7f31863969a4bda9e128166c15ac1ad" member_type: VOTER last_known_addr { host: "127.1.170.130" port: 35035 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 20:09:51.061390 9840 mvcc.cc:205] Tried to move back new op lower bound from 7332635406529429504 to 7332635405444530176. Current Snapshot: MvccSnapshot[applied={T|T < 7332635406529429504}]
I20260923 20:09:51.068850 9841 mvcc.cc:205] Tried to move back new op lower bound from 7332635406529429504 to 7332635405444530176. Current Snapshot: MvccSnapshot[applied={T|T < 7332635406529429504}]
I20260923 20:09:51.101295 9876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:09:51.093985 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.170.130:50423 (local address 127.1.170.131:45757)
0923 20:09:51.094191 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0923 20:09:51.094198 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:09:51.095358 (+ 1160us) server_negotiation.cc:407] Connection header received
0923 20:09:51.095594 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:09:51.095599 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:09:51.095697 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:09:51.095816 (+ 119us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 20:09:51.096602 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:09:51.097454 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:09:51.099592 (+ 2138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:09:51.100009 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:09:51.100105 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:09:51.100641 (+ 536us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 20:09:51.100751 (+ 110us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 20:09:51.100817 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0923 20:09:51.100988 (+ 171us) server_negotiation.cc:299] Negotiation successful
0923 20:09:51.101053 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260923 20:09:51.101502 9886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:09:51.094228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.131:45757 (local address 127.1.170.130:50423)
0923 20:09:51.095194 (+ 966us) negotiation.cc:107] Waiting for socket to connect
0923 20:09:51.095220 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0923 20:09:51.095434 (+ 214us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:09:51.096007 (+ 573us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:09:51.096012 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:09:51.096025 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0923 20:09:51.096374 (+ 349us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:09:51.096387 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:09:51.097595 (+ 1208us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:09:51.097602 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:09:51.099455 (+ 1853us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:09:51.099470 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:09:51.100194 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:09:51.100201 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:09:51.100373 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:09:51.100458 (+ 85us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0923 20:09:51.100851 (+ 393us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0923 20:09:51.100868 (+ 17us) client_negotiation.cc:769] Sending connection context
0923 20:09:51.100960 (+ 92us) client_negotiation.cc:240] Negotiation successful
0923 20:09:51.101102 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":732,"thread_start_us":571,"threads_started":1}
I20260923 20:09:51.147064 9314 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50408:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 20:09:51.147469 9314 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50408:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 20:09:51.153017 9314 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6b22557909954a558ccd24b8fbbc4f78: Sending DeleteTablet for 3 replicas of tablet 292b3ffc765a4e348e71cf99035dbef3
I20260923 20:09:51.154227 9591 tablet_service.cc:1514] Processing DeleteTablet for tablet 292b3ffc765a4e348e71cf99035dbef3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 20:09:51 UTC) from {username='oryx', principal='oryx/127.1.170.190@KRBTEST.COM'} at 127.0.0.1:57368
I20260923 20:09:51.154407 9729 tablet_service.cc:1514] Processing DeleteTablet for tablet 292b3ffc765a4e348e71cf99035dbef3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 20:09:51 UTC) from {username='oryx', principal='oryx/127.1.170.190@KRBTEST.COM'} at 127.0.0.1:59158
I20260923 20:09:51.155306 9452 tablet_service.cc:1514] Processing DeleteTablet for tablet 292b3ffc765a4e348e71cf99035dbef3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 20:09:51 UTC) from {username='oryx', principal='oryx/127.1.170.190@KRBTEST.COM'} at 127.0.0.1:47884
I20260923 20:09:51.178267 9892 tablet_replica.cc:330] T 292b3ffc765a4e348e71cf99035dbef3 P c8c21250d251462898a337ffe611517e: stopping tablet replica
I20260923 20:09:51.180203 9892 raft_consensus.cc:2229] T 292b3ffc765a4e348e71cf99035dbef3 P c8c21250d251462898a337ffe611517e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 20:09:51.180878 9891 tablet_replica.cc:330] T 292b3ffc765a4e348e71cf99035dbef3 P 3176a36c35d2480fb87fa57226b3bd7d: stopping tablet replica
I20260923 20:09:51.181263 9892 raft_consensus.cc:2258] T 292b3ffc765a4e348e71cf99035dbef3 P c8c21250d251462898a337ffe611517e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 20:09:51.181566 9891 raft_consensus.cc:2229] T 292b3ffc765a4e348e71cf99035dbef3 P 3176a36c35d2480fb87fa57226b3bd7d [term 1 LEADER]: Raft consensus shutting down.
I20260923 20:09:51.182647 9891 raft_consensus.cc:2258] T 292b3ffc765a4e348e71cf99035dbef3 P 3176a36c35d2480fb87fa57226b3bd7d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 20:09:51.184763 9892 ts_tablet_manager.cc:1905] T 292b3ffc765a4e348e71cf99035dbef3 P c8c21250d251462898a337ffe611517e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 20:09:51.186060 9891 ts_tablet_manager.cc:1905] T 292b3ffc765a4e348e71cf99035dbef3 P 3176a36c35d2480fb87fa57226b3bd7d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 20:09:51.186682 9890 tablet_replica.cc:330] T 292b3ffc765a4e348e71cf99035dbef3 P f7f31863969a4bda9e128166c15ac1ad: stopping tablet replica
I20260923 20:09:51.186885 9890 raft_consensus.cc:2229] T 292b3ffc765a4e348e71cf99035dbef3 P f7f31863969a4bda9e128166c15ac1ad [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 20:09:51.187165 9890 raft_consensus.cc:2258] T 292b3ffc765a4e348e71cf99035dbef3 P f7f31863969a4bda9e128166c15ac1ad [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 20:09:51.190702 9890 ts_tablet_manager.cc:1905] T 292b3ffc765a4e348e71cf99035dbef3 P f7f31863969a4bda9e128166c15ac1ad: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 20:09:51.195482 9892 ts_tablet_manager.cc:1918] T 292b3ffc765a4e348e71cf99035dbef3 P c8c21250d251462898a337ffe611517e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260923 20:09:51.195693 9892 log.cc:1196] T 292b3ffc765a4e348e71cf99035dbef3 P c8c21250d251462898a337ffe611517e: Deleting WAL directory at /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790194121577027-1706-0/minicluster-data/ts-0/wal/wals/292b3ffc765a4e348e71cf99035dbef3
I20260923 20:09:51.196211 9892 ts_tablet_manager.cc:1939] T 292b3ffc765a4e348e71cf99035dbef3 P c8c21250d251462898a337ffe611517e: Deleting consensus metadata
I20260923 20:09:51.197453 9299 catalog_manager.cc:4841] TS c8c21250d251462898a337ffe611517e (127.1.170.129:34387): tablet 292b3ffc765a4e348e71cf99035dbef3 (table test-table [id=4a527c130a1744edba000c0bb44df839]) successfully deleted
I20260923 20:09:51.203735 9890 ts_tablet_manager.cc:1918] T 292b3ffc765a4e348e71cf99035dbef3 P f7f31863969a4bda9e128166c15ac1ad: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260923 20:09:51.208119 9891 ts_tablet_manager.cc:1918] T 292b3ffc765a4e348e71cf99035dbef3 P 3176a36c35d2480fb87fa57226b3bd7d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260923 20:09:51.208397 9891 log.cc:1196] T 292b3ffc765a4e348e71cf99035dbef3 P 3176a36c35d2480fb87fa57226b3bd7d: Deleting WAL directory at /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790194121577027-1706-0/minicluster-data/ts-2/wal/wals/292b3ffc765a4e348e71cf99035dbef3
I20260923 20:09:51.208961 9891 ts_tablet_manager.cc:1939] T 292b3ffc765a4e348e71cf99035dbef3 P 3176a36c35d2480fb87fa57226b3bd7d: Deleting consensus metadata
I20260923 20:09:51.210310 9301 catalog_manager.cc:4841] TS 3176a36c35d2480fb87fa57226b3bd7d (127.1.170.131:45757): tablet 292b3ffc765a4e348e71cf99035dbef3 (table test-table [id=4a527c130a1744edba000c0bb44df839]) successfully deleted
I20260923 20:09:51.210717 9890 log.cc:1196] T 292b3ffc765a4e348e71cf99035dbef3 P f7f31863969a4bda9e128166c15ac1ad: Deleting WAL directory at /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790194121577027-1706-0/minicluster-data/ts-1/wal/wals/292b3ffc765a4e348e71cf99035dbef3
I20260923 20:09:51.211344 9890 ts_tablet_manager.cc:1939] T 292b3ffc765a4e348e71cf99035dbef3 P f7f31863969a4bda9e128166c15ac1ad: Deleting consensus metadata
I20260923 20:09:51.212766 9299 catalog_manager.cc:4841] TS f7f31863969a4bda9e128166c15ac1ad (127.1.170.130:35035): tablet 292b3ffc765a4e348e71cf99035dbef3 (table test-table [id=4a527c130a1744edba000c0bb44df839]) successfully deleted
Sep 23 20:09:51 dist-test-slave-th4g krb5kdc[9270](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194191, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 23 20:09:51 dist-test-slave-th4g krb5kdc[9270](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.1.170.190@KRBTEST.COM, Server not found in Kerberos database
Sep 23 20:09:51 dist-test-slave-th4g krb5kdc[9270](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.1.170.190@KRBTEST.COM, Server not found in Kerberos database
W20260923 20:09:51.266683 9864 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 20:09:51.254540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50426 (local address 127.1.170.190:45507)
0923 20:09:51.254728 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0923 20:09:51.254734 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:09:51.255433 (+ 699us) server_negotiation.cc:407] Connection header received
0923 20:09:51.255664 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:09:51.255669 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:09:51.255745 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:09:51.255867 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:09:51.257179 (+ 1312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:09:51.258053 (+ 874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:09:51.259288 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:09:51.259752 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:09:51.259945 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:09:51.266490 (+ 6545us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50426: BlockingRecv error: recv got EOF from 127.0.0.1:50426 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Sep 23 20:09:51 dist-test-slave-th4g krb5kdc[9270](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194191, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.1.170.190@KRBTEST.COM
I20260923 20:09:51.291625 9864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:09:51.272381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50442 (local address 127.1.170.190:45507)
0923 20:09:51.272603 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0923 20:09:51.272609 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:09:51.272904 (+ 295us) server_negotiation.cc:407] Connection header received
0923 20:09:51.273104 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:09:51.273110 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:09:51.273198 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:09:51.273325 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:09:51.274991 (+ 1666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:09:51.276300 (+ 1309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:09:51.278268 (+ 1968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:09:51.278935 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:09:51.279066 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:09:51.284091 (+ 5025us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:09:51.284099 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:09:51.284105 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:09:51.284157 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:09:51.288380 (+ 4223us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:09:51.289362 (+ 982us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:09:51.289368 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:09:51.289372 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:09:51.289536 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:09:51.290051 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:09:51.290057 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:09:51.290061 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:09:51.290417 (+ 356us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:09:51.290691 (+ 274us) server_negotiation.cc:1035] Waiting for connection context
0923 20:09:51.291002 (+ 311us) server_negotiation.cc:299] Negotiation successful
0923 20:09:51.291239 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260923 20:09:51.295971 9314 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50442:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 20:09:51.296748 9314 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 20:09:51.305332 9591 tablet_service.cc:1467] Processing CreateTablet for tablet a53e22e7f55041e8a197f873726d3db1 (DEFAULT_TABLE table=test-table [id=0c9b1b02fd1b459cb4658bdf52784031]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 20:09:51.305352 9452 tablet_service.cc:1467] Processing CreateTablet for tablet a53e22e7f55041e8a197f873726d3db1 (DEFAULT_TABLE table=test-table [id=0c9b1b02fd1b459cb4658bdf52784031]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 20:09:51.305804 9452 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a53e22e7f55041e8a197f873726d3db1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:09:51.305809 9591 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a53e22e7f55041e8a197f873726d3db1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:09:51.312373 9826 tablet_bootstrap.cc:492] T a53e22e7f55041e8a197f873726d3db1 P c8c21250d251462898a337ffe611517e: Bootstrap starting.
I20260923 20:09:51.313155 9729 tablet_service.cc:1467] Processing CreateTablet for tablet a53e22e7f55041e8a197f873726d3db1 (DEFAULT_TABLE table=test-table [id=0c9b1b02fd1b459cb4658bdf52784031]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 20:09:51.313270 9828 tablet_bootstrap.cc:492] T a53e22e7f55041e8a197f873726d3db1 P f7f31863969a4bda9e128166c15ac1ad: Bootstrap starting.
I20260923 20:09:51.313603 9729 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a53e22e7f55041e8a197f873726d3db1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:09:51.314071 9826 tablet_bootstrap.cc:654] T a53e22e7f55041e8a197f873726d3db1 P c8c21250d251462898a337ffe611517e: Neither blocks nor log segments found. Creating new log.
I20260923 20:09:51.314977 9828 tablet_bootstrap.cc:654] T a53e22e7f55041e8a197f873726d3db1 P f7f31863969a4bda9e128166c15ac1ad: Neither blocks nor log segments found. Creating new log.
I20260923 20:09:51.323560 9825 tablet_bootstrap.cc:492] T a53e22e7f55041e8a197f873726d3db1 P 3176a36c35d2480fb87fa57226b3bd7d: Bootstrap starting.
I20260923 20:09:51.325575 9828 tablet_bootstrap.cc:492] T a53e22e7f55041e8a197f873726d3db1 P f7f31863969a4bda9e128166c15ac1ad: No bootstrap required, opened a new log
I20260923 20:09:51.325752 9828 ts_tablet_manager.cc:1397] T a53e22e7f55041e8a197f873726d3db1 P f7f31863969a4bda9e128166c15ac1ad: Time spent bootstrapping tablet: real 0.013s user 0.009s sys 0.000s
I20260923 20:09:51.325932 9825 tablet_bootstrap.cc:654] T a53e22e7f55041e8a197f873726d3db1 P 3176a36c35d2480fb87fa57226b3bd7d: Neither blocks nor log segments found. Creating new log.
I20260923 20:09:51.326642 9828 raft_consensus.cc:348] T a53e22e7f55041e8a197f873726d3db1 P f7f31863969a4bda9e128166c15ac1ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8c21250d251462898a337ffe611517e" member_type: VOTER last_known_addr { host: "127.1.170.129" port: 34387 } } peers { permanent_uuid: "f7f31863969a4bda9e128166c15ac1ad" member_type: VOTER last_known_addr { host: "127.1.170.130" port: 35035 } } peers { permanent_uuid: "3176a36c35d2480fb87fa57226b3bd7d" member_type: VOTER last_known_addr { host: "127.1.170.131" port: 45757 } }
I20260923 20:09:51.326906 9828 raft_consensus.cc:374] T a53e22e7f55041e8a197f873726d3db1 P f7f31863969a4bda9e128166c15ac1ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:09:51.327018 9828 raft_consensus.cc:729] T a53e22e7f55041e8a197f873726d3db1 P f7f31863969a4bda9e128166c15ac1ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f7f31863969a4bda9e128166c15ac1ad, State: Initialized, Role: FOLLOWER
I20260923 20:09:51.327378 9828 consensus_queue.cc:260] T a53e22e7f55041e8a197f873726d3db1 P f7f31863969a4bda9e128166c15ac1ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8c21250d251462898a337ffe611517e" member_type: VOTER last_known_addr { host: "127.1.170.129" port: 34387 } } peers { permanent_uuid: "f7f31863969a4bda9e128166c15ac1ad" member_type: VOTER last_known_addr { host: "127.1.170.130" port: 35035 } } peers { permanent_uuid: "3176a36c35d2480fb87fa57226b3bd7d" member_type: VOTER last_known_addr { host: "127.1.170.131" port: 45757 } }
I20260923 20:09:51.330410 9825 tablet_bootstrap.cc:492] T a53e22e7f55041e8a197f873726d3db1 P 3176a36c35d2480fb87fa57226b3bd7d: No bootstrap required, opened a new log
I20260923 20:09:51.330693 9825 ts_tablet_manager.cc:1397] T a53e22e7f55041e8a197f873726d3db1 P 3176a36c35d2480fb87fa57226b3bd7d: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.004s
I20260923 20:09:51.331581 9825 raft_consensus.cc:348] T a53e22e7f55041e8a197f873726d3db1 P 3176a36c35d2480fb87fa57226b3bd7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8c21250d251462898a337ffe611517e" member_type: VOTER last_known_addr { host: "127.1.170.129" port: 34387 } } peers { permanent_uuid: "f7f31863969a4bda9e128166c15ac1ad" member_type: VOTER last_known_addr { host: "127.1.170.130" port: 35035 } } peers { permanent_uuid: "3176a36c35d2480fb87fa57226b3bd7d" member_type: VOTER last_known_addr { host: "127.1.170.131" port: 45757 } }
I20260923 20:09:51.332405 9828 ts_tablet_manager.cc:1428] T a53e22e7f55041e8a197f873726d3db1 P f7f31863969a4bda9e128166c15ac1ad: Time spent starting tablet: real 0.007s user 0.003s sys 0.000s
I20260923 20:09:51.333205 9825 raft_consensus.cc:374] T a53e22e7f55041e8a197f873726d3db1 P 3176a36c35d2480fb87fa57226b3bd7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:09:51.333400 9825 raft_consensus.cc:729] T a53e22e7f55041e8a197f873726d3db1 P 3176a36c35d2480fb87fa57226b3bd7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3176a36c35d2480fb87fa57226b3bd7d, State: Initialized, Role: FOLLOWER
I20260923 20:09:51.333827 9825 consensus_queue.cc:260] T a53e22e7f55041e8a197f873726d3db1 P 3176a36c35d2480fb87fa57226b3bd7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8c21250d251462898a337ffe611517e" member_type: VOTER last_known_addr { host: "127.1.170.129" port: 34387 } } peers { permanent_uuid: "f7f31863969a4bda9e128166c15ac1ad" member_type: VOTER last_known_addr { host: "127.1.170.130" port: 35035 } } peers { permanent_uuid: "3176a36c35d2480fb87fa57226b3bd7d" member_type: VOTER last_known_addr { host: "127.1.170.131" port: 45757 } }
I20260923 20:09:51.335850 9825 ts_tablet_manager.cc:1428] T a53e22e7f55041e8a197f873726d3db1 P 3176a36c35d2480fb87fa57226b3bd7d: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20260923 20:09:51.341140 9826 tablet_bootstrap.cc:492] T a53e22e7f55041e8a197f873726d3db1 P c8c21250d251462898a337ffe611517e: No bootstrap required, opened a new log
I20260923 20:09:51.341300 9826 ts_tablet_manager.cc:1397] T a53e22e7f55041e8a197f873726d3db1 P c8c21250d251462898a337ffe611517e: Time spent bootstrapping tablet: real 0.029s user 0.011s sys 0.000s
I20260923 20:09:51.342147 9826 raft_consensus.cc:348] T a53e22e7f55041e8a197f873726d3db1 P c8c21250d251462898a337ffe611517e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8c21250d251462898a337ffe611517e" member_type: VOTER last_known_addr { host: "127.1.170.129" port: 34387 } } peers { permanent_uuid: "f7f31863969a4bda9e128166c15ac1ad" member_type: VOTER last_known_addr { host: "127.1.170.130" port: 35035 } } peers { permanent_uuid: "3176a36c35d2480fb87fa57226b3bd7d" member_type: VOTER last_known_addr { host: "127.1.170.131" port: 45757 } }
I20260923 20:09:51.342420 9826 raft_consensus.cc:374] T a53e22e7f55041e8a197f873726d3db1 P c8c21250d251462898a337ffe611517e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:09:51.342509 9826 raft_consensus.cc:729] T a53e22e7f55041e8a197f873726d3db1 P c8c21250d251462898a337ffe611517e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8c21250d251462898a337ffe611517e, State: Initialized, Role: FOLLOWER
I20260923 20:09:51.342725 9826 consensus_queue.cc:260] T a53e22e7f55041e8a197f873726d3db1 P c8c21250d251462898a337ffe611517e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8c21250d251462898a337ffe611517e" member_type: VOTER last_known_addr { host: "127.1.170.129" port: 34387 } } peers { permanent_uuid: "f7f31863969a4bda9e128166c15ac1ad" member_type: VOTER last_known_addr { host: "127.1.170.130" port: 35035 } } peers { permanent_uuid: "3176a36c35d2480fb87fa57226b3bd7d" member_type: VOTER last_known_addr { host: "127.1.170.131" port: 45757 } }
I20260923 20:09:51.343307 9826 ts_tablet_manager.cc:1428] T a53e22e7f55041e8a197f873726d3db1 P c8c21250d251462898a337ffe611517e: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260923 20:09:51.517465 9832 raft_consensus.cc:482] T a53e22e7f55041e8a197f873726d3db1 P c8c21250d251462898a337ffe611517e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 20:09:51.517688 9832 raft_consensus.cc:504] T a53e22e7f55041e8a197f873726d3db1 P c8c21250d251462898a337ffe611517e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8c21250d251462898a337ffe611517e" member_type: VOTER last_known_addr { host: "127.1.170.129" port: 34387 } } peers { permanent_uuid: "f7f31863969a4bda9e128166c15ac1ad" member_type: VOTER last_known_addr { host: "127.1.170.130" port: 35035 } } peers { permanent_uuid: "3176a36c35d2480fb87fa57226b3bd7d" member_type: VOTER last_known_addr { host: "127.1.170.131" port: 45757 } }
I20260923 20:09:51.519448 9832 leader_election.cc:288] T a53e22e7f55041e8a197f873726d3db1 P c8c21250d251462898a337ffe611517e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f7f31863969a4bda9e128166c15ac1ad (127.1.170.130:35035), 3176a36c35d2480fb87fa57226b3bd7d (127.1.170.131:45757)
I20260923 20:09:51.529459 9824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:09:51.520255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.170.129:42201 (local address 127.1.170.130:35035)
0923 20:09:51.520453 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0923 20:09:51.520460 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:09:51.521392 (+ 932us) server_negotiation.cc:407] Connection header received
0923 20:09:51.521703 (+ 311us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:09:51.521709 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:09:51.521798 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:09:51.521862 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:09:51.522721 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:09:51.523870 (+ 1149us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:09:51.526720 (+ 2850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:09:51.528099 (+ 1379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:09:51.528274 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:09:51.528364 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0923 20:09:51.529013 (+ 649us) server_negotiation.cc:299] Negotiation successful
0923 20:09:51.529162 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260923 20:09:51.530306 9908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:09:51.519806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.130:35035 (local address 127.1.170.129:42201)
0923 20:09:51.521284 (+ 1478us) negotiation.cc:107] Waiting for socket to connect
0923 20:09:51.521305 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0923 20:09:51.521509 (+ 204us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:09:51.522119 (+ 610us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:09:51.522124 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:09:51.522150 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:09:51.522512 (+ 362us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:09:51.522525 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:09:51.524077 (+ 1552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:09:51.524083 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:09:51.526502 (+ 2419us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:09:51.526520 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:09:51.528570 (+ 2050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:09:51.528576 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:09:51.528809 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:09:51.528847 (+ 38us) client_negotiation.cc:769] Sending connection context
0923 20:09:51.529741 (+ 894us) client_negotiation.cc:240] Negotiation successful
0923 20:09:51.529931 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1352,"thread_start_us":1233,"threads_started":1}
I20260923 20:09:51.531044 9610 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a53e22e7f55041e8a197f873726d3db1" candidate_uuid: "c8c21250d251462898a337ffe611517e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f7f31863969a4bda9e128166c15ac1ad" is_pre_election: true
I20260923 20:09:51.531368 9610 raft_consensus.cc:2454] T a53e22e7f55041e8a197f873726d3db1 P f7f31863969a4bda9e128166c15ac1ad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c8c21250d251462898a337ffe611517e in term 0.
I20260923 20:09:51.532002 9402 leader_election.cc:302] T a53e22e7f55041e8a197f873726d3db1 P c8c21250d251462898a337ffe611517e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c8c21250d251462898a337ffe611517e, f7f31863969a4bda9e128166c15ac1ad; no voters:
I20260923 20:09:51.532492 9832 raft_consensus.cc:2781] T a53e22e7f55041e8a197f873726d3db1 P c8c21250d251462898a337ffe611517e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 20:09:51.532632 9832 raft_consensus.cc:482] T a53e22e7f55041e8a197f873726d3db1 P c8c21250d251462898a337ffe611517e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 20:09:51.532734 9832 raft_consensus.cc:3037] T a53e22e7f55041e8a197f873726d3db1 P c8c21250d251462898a337ffe611517e [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:09:51.536233 9832 raft_consensus.cc:504] T a53e22e7f55041e8a197f873726d3db1 P c8c21250d251462898a337ffe611517e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8c21250d251462898a337ffe611517e" member_type: VOTER last_known_addr { host: "127.1.170.129" port: 34387 } } peers { permanent_uuid: "f7f31863969a4bda9e128166c15ac1ad" member_type: VOTER last_known_addr { host: "127.1.170.130" port: 35035 } } peers { permanent_uuid: "3176a36c35d2480fb87fa57226b3bd7d" member_type: VOTER last_known_addr { host: "127.1.170.131" port: 45757 } }
I20260923 20:09:51.537685 9610 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a53e22e7f55041e8a197f873726d3db1" candidate_uuid: "c8c21250d251462898a337ffe611517e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f7f31863969a4bda9e128166c15ac1ad"
I20260923 20:09:51.537936 9610 raft_consensus.cc:3037] T a53e22e7f55041e8a197f873726d3db1 P f7f31863969a4bda9e128166c15ac1ad [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:09:51.541651 9876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:09:51.520723 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.170.129:36943 (local address 127.1.170.131:45757)
0923 20:09:51.520931 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0923 20:09:51.520942 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0923 20:09:51.533507 (+ 12565us) server_negotiation.cc:407] Connection header received
0923 20:09:51.534169 (+ 662us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:09:51.534175 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:09:51.534247 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:09:51.534306 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:09:51.535182 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:09:51.536413 (+ 1231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:09:51.538992 (+ 2579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:09:51.540329 (+ 1337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:09:51.540460 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:09:51.540530 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0923 20:09:51.541201 (+ 671us) server_negotiation.cc:299] Negotiation successful
0923 20:09:51.541360 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260923 20:09:51.542348 9610 raft_consensus.cc:2454] T a53e22e7f55041e8a197f873726d3db1 P f7f31863969a4bda9e128166c15ac1ad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c8c21250d251462898a337ffe611517e in term 1.
I20260923 20:09:51.542222 9832 leader_election.cc:288] T a53e22e7f55041e8a197f873726d3db1 P c8c21250d251462898a337ffe611517e [CANDIDATE]: Term 1 election: Requested vote from peers f7f31863969a4bda9e128166c15ac1ad (127.1.170.130:35035), 3176a36c35d2480fb87fa57226b3bd7d (127.1.170.131:45757)
I20260923 20:09:51.543028 9909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:09:51.520522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.131:45757 (local address 127.1.170.129:36943)
0923 20:09:51.533376 (+ 12854us) negotiation.cc:107] Waiting for socket to connect
0923 20:09:51.533397 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0923 20:09:51.533991 (+ 594us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:09:51.534490 (+ 499us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:09:51.534494 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:09:51.534519 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:09:51.534924 (+ 405us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:09:51.534935 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:09:51.536598 (+ 1663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:09:51.536603 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:09:51.538789 (+ 2186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:09:51.538807 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:09:51.540774 (+ 1967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:09:51.540780 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:09:51.540986 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:09:51.541025 (+ 39us) client_negotiation.cc:769] Sending connection context
0923 20:09:51.542134 (+ 1109us) client_negotiation.cc:240] Negotiation successful
0923 20:09:51.542306 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12692,"thread_start_us":13036,"threads_started":1}
I20260923 20:09:51.543088 9402 leader_election.cc:302] T a53e22e7f55041e8a197f873726d3db1 P c8c21250d251462898a337ffe611517e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c8c21250d251462898a337ffe611517e, f7f31863969a4bda9e128166c15ac1ad; no voters:
I20260923 20:09:51.543643 9832 raft_consensus.cc:2781] T a53e22e7f55041e8a197f873726d3db1 P c8c21250d251462898a337ffe611517e [term 1 FOLLOWER]: Leader election won for term 1
I20260923 20:09:51.543745 9755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a53e22e7f55041e8a197f873726d3db1" candidate_uuid: "c8c21250d251462898a337ffe611517e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3176a36c35d2480fb87fa57226b3bd7d" is_pre_election: true
I20260923 20:09:51.543872 9754 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a53e22e7f55041e8a197f873726d3db1" candidate_uuid: "c8c21250d251462898a337ffe611517e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3176a36c35d2480fb87fa57226b3bd7d"
I20260923 20:09:51.543999 9755 raft_consensus.cc:2454] T a53e22e7f55041e8a197f873726d3db1 P 3176a36c35d2480fb87fa57226b3bd7d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c8c21250d251462898a337ffe611517e in term 0.
I20260923 20:09:51.554271 9832 raft_consensus.cc:686] T a53e22e7f55041e8a197f873726d3db1 P c8c21250d251462898a337ffe611517e [term 1 LEADER]: Becoming Leader. State: Replica: c8c21250d251462898a337ffe611517e, State: Running, Role: LEADER
I20260923 20:09:51.554689 9832 consensus_queue.cc:237] T a53e22e7f55041e8a197f873726d3db1 P c8c21250d251462898a337ffe611517e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8c21250d251462898a337ffe611517e" member_type: VOTER last_known_addr { host: "127.1.170.129" port: 34387 } } peers { permanent_uuid: "f7f31863969a4bda9e128166c15ac1ad" member_type: VOTER last_known_addr { host: "127.1.170.130" port: 35035 } } peers { permanent_uuid: "3176a36c35d2480fb87fa57226b3bd7d" member_type: VOTER last_known_addr { host: "127.1.170.131" port: 45757 } }
I20260923 20:09:51.558305 9315 catalog_manager.cc:5495] T a53e22e7f55041e8a197f873726d3db1 P c8c21250d251462898a337ffe611517e reported cstate change: term changed from 0 to 1, leader changed from <none> to c8c21250d251462898a337ffe611517e (127.1.170.129). New cstate: current_term: 1 leader_uuid: "c8c21250d251462898a337ffe611517e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8c21250d251462898a337ffe611517e" member_type: VOTER last_known_addr { host: "127.1.170.129" port: 34387 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f7f31863969a4bda9e128166c15ac1ad" member_type: VOTER last_known_addr { host: "127.1.170.130" port: 35035 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3176a36c35d2480fb87fa57226b3bd7d" member_type: VOTER last_known_addr { host: "127.1.170.131" port: 45757 } health_report { overall_health: UNKNOWN } } }
I20260923 20:09:51.628860 9913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:09:51.622541 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47896 (local address 127.1.170.129:34387)
0923 20:09:51.623082 (+ 541us) server_negotiation.cc:206] Beginning negotiation
0923 20:09:51.623086 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 20:09:51.623103 (+ 17us) server_negotiation.cc:407] Connection header received
0923 20:09:51.623203 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:09:51.623207 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:09:51.623282 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:09:51.623464 (+ 182us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 20:09:51.623905 (+ 441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:09:51.624835 (+ 930us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:09:51.626456 (+ 1621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:09:51.626877 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:09:51.627007 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:09:51.627455 (+ 448us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 20:09:51.628152 (+ 697us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 20:09:51.628317 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0923 20:09:51.628398 (+ 81us) server_negotiation.cc:299] Negotiation successful
0923 20:09:51.628476 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":425,"thread_start_us":263,"threads_started":1}
I20260923 20:09:51.633045 9610 raft_consensus.cc:1261] T a53e22e7f55041e8a197f873726d3db1 P f7f31863969a4bda9e128166c15ac1ad [term 1 FOLLOWER]: Refusing update from remote peer c8c21250d251462898a337ffe611517e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 20:09:51.633339 9755 raft_consensus.cc:3037] T a53e22e7f55041e8a197f873726d3db1 P 3176a36c35d2480fb87fa57226b3bd7d [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:09:51.633901 9910 consensus_queue.cc:1035] T a53e22e7f55041e8a197f873726d3db1 P c8c21250d251462898a337ffe611517e [LEADER]: Connected to new peer: Peer: permanent_uuid: "f7f31863969a4bda9e128166c15ac1ad" member_type: VOTER last_known_addr { host: "127.1.170.130" port: 35035 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 20:09:51.637028 9755 raft_consensus.cc:1261] T a53e22e7f55041e8a197f873726d3db1 P 3176a36c35d2480fb87fa57226b3bd7d [term 1 FOLLOWER]: Refusing update from remote peer c8c21250d251462898a337ffe611517e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 20:09:51.637818 9910 consensus_queue.cc:1035] T a53e22e7f55041e8a197f873726d3db1 P c8c21250d251462898a337ffe611517e [LEADER]: Connected to new peer: Peer: permanent_uuid: "3176a36c35d2480fb87fa57226b3bd7d" member_type: VOTER last_known_addr { host: "127.1.170.131" port: 45757 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 20:09:51.663403 9315 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50442:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 20:09:51.663702 9315 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50442:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 20:09:51.667325 9315 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6b22557909954a558ccd24b8fbbc4f78: Sending DeleteTablet for 3 replicas of tablet a53e22e7f55041e8a197f873726d3db1
I20260923 20:09:51.668376 9452 tablet_service.cc:1514] Processing DeleteTablet for tablet a53e22e7f55041e8a197f873726d3db1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 20:09:51 UTC) from {username='oryx', principal='oryx/127.1.170.190@KRBTEST.COM'} at 127.0.0.1:47884
I20260923 20:09:51.668483 9729 tablet_service.cc:1514] Processing DeleteTablet for tablet a53e22e7f55041e8a197f873726d3db1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 20:09:51 UTC) from {username='oryx', principal='oryx/127.1.170.190@KRBTEST.COM'} at 127.0.0.1:59158
I20260923 20:09:51.668660 9892 tablet_replica.cc:330] T a53e22e7f55041e8a197f873726d3db1 P c8c21250d251462898a337ffe611517e: stopping tablet replica
I20260923 20:09:51.668740 9891 tablet_replica.cc:330] T a53e22e7f55041e8a197f873726d3db1 P 3176a36c35d2480fb87fa57226b3bd7d: stopping tablet replica
I20260923 20:09:51.668869 9892 raft_consensus.cc:2229] T a53e22e7f55041e8a197f873726d3db1 P c8c21250d251462898a337ffe611517e [term 1 LEADER]: Raft consensus shutting down.
I20260923 20:09:51.668948 9891 raft_consensus.cc:2229] T a53e22e7f55041e8a197f873726d3db1 P 3176a36c35d2480fb87fa57226b3bd7d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 20:09:51.669126 9891 raft_consensus.cc:2258] T a53e22e7f55041e8a197f873726d3db1 P 3176a36c35d2480fb87fa57226b3bd7d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 20:09:51.669345 9892 raft_consensus.cc:2258] T a53e22e7f55041e8a197f873726d3db1 P c8c21250d251462898a337ffe611517e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 20:09:51.670315 9891 ts_tablet_manager.cc:1905] T a53e22e7f55041e8a197f873726d3db1 P 3176a36c35d2480fb87fa57226b3bd7d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 20:09:51.670308 9591 tablet_service.cc:1514] Processing DeleteTablet for tablet a53e22e7f55041e8a197f873726d3db1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 20:09:51 UTC) from {username='oryx', principal='oryx/127.1.170.190@KRBTEST.COM'} at 127.0.0.1:57368
I20260923 20:09:51.670560 9890 tablet_replica.cc:330] T a53e22e7f55041e8a197f873726d3db1 P f7f31863969a4bda9e128166c15ac1ad: stopping tablet replica
I20260923 20:09:51.670773 9890 raft_consensus.cc:2229] T a53e22e7f55041e8a197f873726d3db1 P f7f31863969a4bda9e128166c15ac1ad [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 20:09:51.670990 9890 raft_consensus.cc:2258] T a53e22e7f55041e8a197f873726d3db1 P f7f31863969a4bda9e128166c15ac1ad [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 20:09:51.671809 9315 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50442:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260923 20:09:51.672079 9892 ts_tablet_manager.cc:1905] T a53e22e7f55041e8a197f873726d3db1 P c8c21250d251462898a337ffe611517e: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260923 20:09:51.672434 9315 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 20:09:51.676172 9890 ts_tablet_manager.cc:1905] T a53e22e7f55041e8a197f873726d3db1 P f7f31863969a4bda9e128166c15ac1ad: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 20:09:51.678247 9892 ts_tablet_manager.cc:1918] T a53e22e7f55041e8a197f873726d3db1 P c8c21250d251462898a337ffe611517e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 20:09:51.678403 9892 log.cc:1196] T a53e22e7f55041e8a197f873726d3db1 P c8c21250d251462898a337ffe611517e: Deleting WAL directory at /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790194121577027-1706-0/minicluster-data/ts-0/wal/wals/a53e22e7f55041e8a197f873726d3db1
I20260923 20:09:51.678829 9892 ts_tablet_manager.cc:1939] T a53e22e7f55041e8a197f873726d3db1 P c8c21250d251462898a337ffe611517e: Deleting consensus metadata
I20260923 20:09:51.678846 9891 ts_tablet_manager.cc:1918] T a53e22e7f55041e8a197f873726d3db1 P 3176a36c35d2480fb87fa57226b3bd7d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 20:09:51.679121 9891 log.cc:1196] T a53e22e7f55041e8a197f873726d3db1 P 3176a36c35d2480fb87fa57226b3bd7d: Deleting WAL directory at /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790194121577027-1706-0/minicluster-data/ts-2/wal/wals/a53e22e7f55041e8a197f873726d3db1
I20260923 20:09:51.679590 9891 ts_tablet_manager.cc:1939] T a53e22e7f55041e8a197f873726d3db1 P 3176a36c35d2480fb87fa57226b3bd7d: Deleting consensus metadata
I20260923 20:09:51.680903 9301 catalog_manager.cc:4841] TS 3176a36c35d2480fb87fa57226b3bd7d (127.1.170.131:45757): tablet a53e22e7f55041e8a197f873726d3db1 (table test-table [id=0c9b1b02fd1b459cb4658bdf52784031]) successfully deleted
I20260923 20:09:51.681173 9299 catalog_manager.cc:4841] TS c8c21250d251462898a337ffe611517e (127.1.170.129:34387): tablet a53e22e7f55041e8a197f873726d3db1 (table test-table [id=0c9b1b02fd1b459cb4658bdf52784031]) successfully deleted
I20260923 20:09:51.684924 9729 tablet_service.cc:1467] Processing CreateTablet for tablet 8e5a31a539924474b31545a7c6737425 (DEFAULT_TABLE table=test-table [id=51f3999106704c69b05b1b8a8b019a3b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 20:09:51.685231 9890 ts_tablet_manager.cc:1918] T a53e22e7f55041e8a197f873726d3db1 P f7f31863969a4bda9e128166c15ac1ad: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 20:09:51.685364 9729 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8e5a31a539924474b31545a7c6737425. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:09:51.685374 9890 log.cc:1196] T a53e22e7f55041e8a197f873726d3db1 P f7f31863969a4bda9e128166c15ac1ad: Deleting WAL directory at /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790194121577027-1706-0/minicluster-data/ts-1/wal/wals/a53e22e7f55041e8a197f873726d3db1
I20260923 20:09:51.685762 9890 ts_tablet_manager.cc:1939] T a53e22e7f55041e8a197f873726d3db1 P f7f31863969a4bda9e128166c15ac1ad: Deleting consensus metadata
I20260923 20:09:51.690114 9299 catalog_manager.cc:4841] TS f7f31863969a4bda9e128166c15ac1ad (127.1.170.130:35035): tablet a53e22e7f55041e8a197f873726d3db1 (table test-table [id=0c9b1b02fd1b459cb4658bdf52784031]) successfully deleted
I20260923 20:09:51.690552 9452 tablet_service.cc:1467] Processing CreateTablet for tablet 8e5a31a539924474b31545a7c6737425 (DEFAULT_TABLE table=test-table [id=51f3999106704c69b05b1b8a8b019a3b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 20:09:51.690987 9452 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8e5a31a539924474b31545a7c6737425. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:09:51.691506 9591 tablet_service.cc:1467] Processing CreateTablet for tablet 8e5a31a539924474b31545a7c6737425 (DEFAULT_TABLE table=test-table [id=51f3999106704c69b05b1b8a8b019a3b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 20:09:51.691886 9591 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8e5a31a539924474b31545a7c6737425. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:09:51.698350 9826 tablet_bootstrap.cc:492] T 8e5a31a539924474b31545a7c6737425 P c8c21250d251462898a337ffe611517e: Bootstrap starting.
I20260923 20:09:51.700140 9826 tablet_bootstrap.cc:654] T 8e5a31a539924474b31545a7c6737425 P c8c21250d251462898a337ffe611517e: Neither blocks nor log segments found. Creating new log.
I20260923 20:09:51.701390 9828 tablet_bootstrap.cc:492] T 8e5a31a539924474b31545a7c6737425 P f7f31863969a4bda9e128166c15ac1ad: Bootstrap starting.
I20260923 20:09:51.702524 9825 tablet_bootstrap.cc:492] T 8e5a31a539924474b31545a7c6737425 P 3176a36c35d2480fb87fa57226b3bd7d: Bootstrap starting.
I20260923 20:09:51.702977 9828 tablet_bootstrap.cc:654] T 8e5a31a539924474b31545a7c6737425 P f7f31863969a4bda9e128166c15ac1ad: Neither blocks nor log segments found. Creating new log.
I20260923 20:09:51.704218 9825 tablet_bootstrap.cc:654] T 8e5a31a539924474b31545a7c6737425 P 3176a36c35d2480fb87fa57226b3bd7d: Neither blocks nor log segments found. Creating new log.
I20260923 20:09:51.711092 9826 tablet_bootstrap.cc:492] T 8e5a31a539924474b31545a7c6737425 P c8c21250d251462898a337ffe611517e: No bootstrap required, opened a new log
I20260923 20:09:51.711301 9826 ts_tablet_manager.cc:1397] T 8e5a31a539924474b31545a7c6737425 P c8c21250d251462898a337ffe611517e: Time spent bootstrapping tablet: real 0.013s user 0.007s sys 0.003s
I20260923 20:09:51.712078 9826 raft_consensus.cc:348] T 8e5a31a539924474b31545a7c6737425 P c8c21250d251462898a337ffe611517e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8c21250d251462898a337ffe611517e" member_type: VOTER last_known_addr { host: "127.1.170.129" port: 34387 } } peers { permanent_uuid: "f7f31863969a4bda9e128166c15ac1ad" member_type: VOTER last_known_addr { host: "127.1.170.130" port: 35035 } } peers { permanent_uuid: "3176a36c35d2480fb87fa57226b3bd7d" member_type: VOTER last_known_addr { host: "127.1.170.131" port: 45757 } }
I20260923 20:09:51.712392 9826 raft_consensus.cc:374] T 8e5a31a539924474b31545a7c6737425 P c8c21250d251462898a337ffe611517e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:09:51.712484 9826 raft_consensus.cc:729] T 8e5a31a539924474b31545a7c6737425 P c8c21250d251462898a337ffe611517e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8c21250d251462898a337ffe611517e, State: Initialized, Role: FOLLOWER
I20260923 20:09:51.713140 9826 consensus_queue.cc:260] T 8e5a31a539924474b31545a7c6737425 P c8c21250d251462898a337ffe611517e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8c21250d251462898a337ffe611517e" member_type: VOTER last_known_addr { host: "127.1.170.129" port: 34387 } } peers { permanent_uuid: "f7f31863969a4bda9e128166c15ac1ad" member_type: VOTER last_known_addr { host: "127.1.170.130" port: 35035 } } peers { permanent_uuid: "3176a36c35d2480fb87fa57226b3bd7d" member_type: VOTER last_known_addr { host: "127.1.170.131" port: 45757 } }
I20260923 20:09:51.713822 9826 ts_tablet_manager.cc:1428] T 8e5a31a539924474b31545a7c6737425 P c8c21250d251462898a337ffe611517e: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20260923 20:09:51.719053 9828 tablet_bootstrap.cc:492] T 8e5a31a539924474b31545a7c6737425 P f7f31863969a4bda9e128166c15ac1ad: No bootstrap required, opened a new log
I20260923 20:09:51.719242 9828 ts_tablet_manager.cc:1397] T 8e5a31a539924474b31545a7c6737425 P f7f31863969a4bda9e128166c15ac1ad: Time spent bootstrapping tablet: real 0.018s user 0.008s sys 0.004s
I20260923 20:09:51.719466 9825 tablet_bootstrap.cc:492] T 8e5a31a539924474b31545a7c6737425 P 3176a36c35d2480fb87fa57226b3bd7d: No bootstrap required, opened a new log
I20260923 20:09:51.719671 9825 ts_tablet_manager.cc:1397] T 8e5a31a539924474b31545a7c6737425 P 3176a36c35d2480fb87fa57226b3bd7d: Time spent bootstrapping tablet: real 0.017s user 0.007s sys 0.003s
I20260923 20:09:51.720088 9828 raft_consensus.cc:348] T 8e5a31a539924474b31545a7c6737425 P f7f31863969a4bda9e128166c15ac1ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8c21250d251462898a337ffe611517e" member_type: VOTER last_known_addr { host: "127.1.170.129" port: 34387 } } peers { permanent_uuid: "f7f31863969a4bda9e128166c15ac1ad" member_type: VOTER last_known_addr { host: "127.1.170.130" port: 35035 } } peers { permanent_uuid: "3176a36c35d2480fb87fa57226b3bd7d" member_type: VOTER last_known_addr { host: "127.1.170.131" port: 45757 } }
I20260923 20:09:51.720336 9828 raft_consensus.cc:374] T 8e5a31a539924474b31545a7c6737425 P f7f31863969a4bda9e128166c15ac1ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:09:51.720420 9828 raft_consensus.cc:729] T 8e5a31a539924474b31545a7c6737425 P f7f31863969a4bda9e128166c15ac1ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f7f31863969a4bda9e128166c15ac1ad, State: Initialized, Role: FOLLOWER
I20260923 20:09:51.720608 9825 raft_consensus.cc:348] T 8e5a31a539924474b31545a7c6737425 P 3176a36c35d2480fb87fa57226b3bd7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8c21250d251462898a337ffe611517e" member_type: VOTER last_known_addr { host: "127.1.170.129" port: 34387 } } peers { permanent_uuid: "f7f31863969a4bda9e128166c15ac1ad" member_type: VOTER last_known_addr { host: "127.1.170.130" port: 35035 } } peers { permanent_uuid: "3176a36c35d2480fb87fa57226b3bd7d" member_type: VOTER last_known_addr { host: "127.1.170.131" port: 45757 } }
I20260923 20:09:51.720695 9828 consensus_queue.cc:260] T 8e5a31a539924474b31545a7c6737425 P f7f31863969a4bda9e128166c15ac1ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8c21250d251462898a337ffe611517e" member_type: VOTER last_known_addr { host: "127.1.170.129" port: 34387 } } peers { permanent_uuid: "f7f31863969a4bda9e128166c15ac1ad" member_type: VOTER last_known_addr { host: "127.1.170.130" port: 35035 } } peers { permanent_uuid: "3176a36c35d2480fb87fa57226b3bd7d" member_type: VOTER last_known_addr { host: "127.1.170.131" port: 45757 } }
I20260923 20:09:51.720849 9825 raft_consensus.cc:374] T 8e5a31a539924474b31545a7c6737425 P 3176a36c35d2480fb87fa57226b3bd7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:09:51.720916 9825 raft_consensus.cc:729] T 8e5a31a539924474b31545a7c6737425 P 3176a36c35d2480fb87fa57226b3bd7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3176a36c35d2480fb87fa57226b3bd7d, State: Initialized, Role: FOLLOWER
I20260923 20:09:51.721179 9825 consensus_queue.cc:260] T 8e5a31a539924474b31545a7c6737425 P 3176a36c35d2480fb87fa57226b3bd7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8c21250d251462898a337ffe611517e" member_type: VOTER last_known_addr { host: "127.1.170.129" port: 34387 } } peers { permanent_uuid: "f7f31863969a4bda9e128166c15ac1ad" member_type: VOTER last_known_addr { host: "127.1.170.130" port: 35035 } } peers { permanent_uuid: "3176a36c35d2480fb87fa57226b3bd7d" member_type: VOTER last_known_addr { host: "127.1.170.131" port: 45757 } }
I20260923 20:09:51.721832 9825 ts_tablet_manager.cc:1428] T 8e5a31a539924474b31545a7c6737425 P 3176a36c35d2480fb87fa57226b3bd7d: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260923 20:09:51.722164 9828 ts_tablet_manager.cc:1428] T 8e5a31a539924474b31545a7c6737425 P f7f31863969a4bda9e128166c15ac1ad: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260923 20:09:51.797781 9844 raft_consensus.cc:482] T 8e5a31a539924474b31545a7c6737425 P f7f31863969a4bda9e128166c15ac1ad [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 20:09:51.798066 9844 raft_consensus.cc:504] T 8e5a31a539924474b31545a7c6737425 P f7f31863969a4bda9e128166c15ac1ad [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8c21250d251462898a337ffe611517e" member_type: VOTER last_known_addr { host: "127.1.170.129" port: 34387 } } peers { permanent_uuid: "f7f31863969a4bda9e128166c15ac1ad" member_type: VOTER last_known_addr { host: "127.1.170.130" port: 35035 } } peers { permanent_uuid: "3176a36c35d2480fb87fa57226b3bd7d" member_type: VOTER last_known_addr { host: "127.1.170.131" port: 45757 } }
I20260923 20:09:51.798790 9844 leader_election.cc:288] T 8e5a31a539924474b31545a7c6737425 P f7f31863969a4bda9e128166c15ac1ad [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c8c21250d251462898a337ffe611517e (127.1.170.129:34387), 3176a36c35d2480fb87fa57226b3bd7d (127.1.170.131:45757)
I20260923 20:09:51.799371 9755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e5a31a539924474b31545a7c6737425" candidate_uuid: "f7f31863969a4bda9e128166c15ac1ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3176a36c35d2480fb87fa57226b3bd7d" is_pre_election: true
I20260923 20:09:51.799335 9473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e5a31a539924474b31545a7c6737425" candidate_uuid: "f7f31863969a4bda9e128166c15ac1ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8c21250d251462898a337ffe611517e" is_pre_election: true
I20260923 20:09:51.799611 9755 raft_consensus.cc:2454] T 8e5a31a539924474b31545a7c6737425 P 3176a36c35d2480fb87fa57226b3bd7d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f7f31863969a4bda9e128166c15ac1ad in term 0.
I20260923 20:09:51.799616 9473 raft_consensus.cc:2454] T 8e5a31a539924474b31545a7c6737425 P c8c21250d251462898a337ffe611517e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f7f31863969a4bda9e128166c15ac1ad in term 0.
I20260923 20:09:51.800037 9544 leader_election.cc:302] T 8e5a31a539924474b31545a7c6737425 P f7f31863969a4bda9e128166c15ac1ad [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3176a36c35d2480fb87fa57226b3bd7d, f7f31863969a4bda9e128166c15ac1ad; no voters:
I20260923 20:09:51.800390 9844 raft_consensus.cc:2781] T 8e5a31a539924474b31545a7c6737425 P f7f31863969a4bda9e128166c15ac1ad [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 20:09:51.800552 9844 raft_consensus.cc:482] T 8e5a31a539924474b31545a7c6737425 P f7f31863969a4bda9e128166c15ac1ad [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 20:09:51.800648 9844 raft_consensus.cc:3037] T 8e5a31a539924474b31545a7c6737425 P f7f31863969a4bda9e128166c15ac1ad [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:09:51.803784 9844 raft_consensus.cc:504] T 8e5a31a539924474b31545a7c6737425 P f7f31863969a4bda9e128166c15ac1ad [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8c21250d251462898a337ffe611517e" member_type: VOTER last_known_addr { host: "127.1.170.129" port: 34387 } } peers { permanent_uuid: "f7f31863969a4bda9e128166c15ac1ad" member_type: VOTER last_known_addr { host: "127.1.170.130" port: 35035 } } peers { permanent_uuid: "3176a36c35d2480fb87fa57226b3bd7d" member_type: VOTER last_known_addr { host: "127.1.170.131" port: 45757 } }
I20260923 20:09:51.804579 9844 leader_election.cc:288] T 8e5a31a539924474b31545a7c6737425 P f7f31863969a4bda9e128166c15ac1ad [CANDIDATE]: Term 1 election: Requested vote from peers c8c21250d251462898a337ffe611517e (127.1.170.129:34387), 3176a36c35d2480fb87fa57226b3bd7d (127.1.170.131:45757)
I20260923 20:09:51.805236 9755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e5a31a539924474b31545a7c6737425" candidate_uuid: "f7f31863969a4bda9e128166c15ac1ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3176a36c35d2480fb87fa57226b3bd7d"
I20260923 20:09:51.805281 9473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e5a31a539924474b31545a7c6737425" candidate_uuid: "f7f31863969a4bda9e128166c15ac1ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8c21250d251462898a337ffe611517e"
I20260923 20:09:51.805465 9755 raft_consensus.cc:3037] T 8e5a31a539924474b31545a7c6737425 P 3176a36c35d2480fb87fa57226b3bd7d [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:09:51.805508 9473 raft_consensus.cc:3037] T 8e5a31a539924474b31545a7c6737425 P c8c21250d251462898a337ffe611517e [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:09:51.807948 9473 raft_consensus.cc:2454] T 8e5a31a539924474b31545a7c6737425 P c8c21250d251462898a337ffe611517e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f7f31863969a4bda9e128166c15ac1ad in term 1.
I20260923 20:09:51.808499 9755 raft_consensus.cc:2454] T 8e5a31a539924474b31545a7c6737425 P 3176a36c35d2480fb87fa57226b3bd7d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f7f31863969a4bda9e128166c15ac1ad in term 1.
I20260923 20:09:51.808501 9542 leader_election.cc:302] T 8e5a31a539924474b31545a7c6737425 P f7f31863969a4bda9e128166c15ac1ad [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c8c21250d251462898a337ffe611517e, f7f31863969a4bda9e128166c15ac1ad; no voters:
I20260923 20:09:51.808802 9844 raft_consensus.cc:2781] T 8e5a31a539924474b31545a7c6737425 P f7f31863969a4bda9e128166c15ac1ad [term 1 FOLLOWER]: Leader election won for term 1
I20260923 20:09:51.809511 9844 raft_consensus.cc:686] T 8e5a31a539924474b31545a7c6737425 P f7f31863969a4bda9e128166c15ac1ad [term 1 LEADER]: Becoming Leader. State: Replica: f7f31863969a4bda9e128166c15ac1ad, State: Running, Role: LEADER
I20260923 20:09:51.809821 9844 consensus_queue.cc:237] T 8e5a31a539924474b31545a7c6737425 P f7f31863969a4bda9e128166c15ac1ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8c21250d251462898a337ffe611517e" member_type: VOTER last_known_addr { host: "127.1.170.129" port: 34387 } } peers { permanent_uuid: "f7f31863969a4bda9e128166c15ac1ad" member_type: VOTER last_known_addr { host: "127.1.170.130" port: 35035 } } peers { permanent_uuid: "3176a36c35d2480fb87fa57226b3bd7d" member_type: VOTER last_known_addr { host: "127.1.170.131" port: 45757 } }
I20260923 20:09:51.812531 9314 catalog_manager.cc:5495] T 8e5a31a539924474b31545a7c6737425 P f7f31863969a4bda9e128166c15ac1ad reported cstate change: term changed from 0 to 1, leader changed from <none> to f7f31863969a4bda9e128166c15ac1ad (127.1.170.130). New cstate: current_term: 1 leader_uuid: "f7f31863969a4bda9e128166c15ac1ad" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8c21250d251462898a337ffe611517e" member_type: VOTER last_known_addr { host: "127.1.170.129" port: 34387 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f7f31863969a4bda9e128166c15ac1ad" member_type: VOTER last_known_addr { host: "127.1.170.130" port: 35035 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3176a36c35d2480fb87fa57226b3bd7d" member_type: VOTER last_known_addr { host: "127.1.170.131" port: 45757 } health_report { overall_health: UNKNOWN } } }
I20260923 20:09:51.852380 9824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:09:51.846527 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57400 (local address 127.1.170.130:35035)
0923 20:09:51.847134 (+ 607us) server_negotiation.cc:206] Beginning negotiation
0923 20:09:51.847140 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:09:51.847156 (+ 16us) server_negotiation.cc:407] Connection header received
0923 20:09:51.847214 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:09:51.847220 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:09:51.847297 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:09:51.847387 (+ 90us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 20:09:51.847746 (+ 359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:09:51.848826 (+ 1080us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:09:51.850274 (+ 1448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:09:51.850949 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:09:51.851075 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:09:51.851584 (+ 509us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 20:09:51.851761 (+ 177us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 20:09:51.851854 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0923 20:09:51.852010 (+ 156us) server_negotiation.cc:299] Negotiation successful
0923 20:09:51.852091 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":484}
I20260923 20:09:51.869088 9824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:09:51.862855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.170.130:42609 (local address 127.1.170.130:35035)
0923 20:09:51.863043 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0923 20:09:51.863049 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:09:51.863276 (+ 227us) server_negotiation.cc:407] Connection header received
0923 20:09:51.863468 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:09:51.863473 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:09:51.863546 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:09:51.863644 (+ 98us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 20:09:51.864129 (+ 485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:09:51.865179 (+ 1050us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:09:51.867145 (+ 1966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:09:51.867809 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:09:51.867956 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:09:51.868360 (+ 404us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 20:09:51.868485 (+ 125us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 20:09:51.868569 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0923 20:09:51.868711 (+ 142us) server_negotiation.cc:299] Negotiation successful
0923 20:09:51.868797 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":49}
I20260923 20:09:51.869187 9930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:09:51.862566 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.130:35035 (local address 127.1.170.130:42609)
0923 20:09:51.863164 (+ 598us) negotiation.cc:107] Waiting for socket to connect
0923 20:09:51.863188 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0923 20:09:51.863344 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:09:51.863687 (+ 343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:09:51.863692 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:09:51.863709 (+ 17us) client_negotiation.cc:189] Negotiated authn=TOKEN
0923 20:09:51.863981 (+ 272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:09:51.863993 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:09:51.865323 (+ 1330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:09:51.865329 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:09:51.867004 (+ 1675us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:09:51.867026 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:09:51.867971 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:09:51.867977 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:09:51.868153 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:09:51.868201 (+ 48us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0923 20:09:51.868588 (+ 387us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0923 20:09:51.868599 (+ 11us) client_negotiation.cc:769] Sending connection context
0923 20:09:51.868681 (+ 82us) client_negotiation.cc:240] Negotiation successful
0923 20:09:51.868823 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":449,"thread_start_us":479,"threads_started":1}
I20260923 20:09:51.882704 9754 raft_consensus.cc:1261] T 8e5a31a539924474b31545a7c6737425 P 3176a36c35d2480fb87fa57226b3bd7d [term 1 FOLLOWER]: Refusing update from remote peer f7f31863969a4bda9e128166c15ac1ad: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 20:09:51.884235 9926 consensus_queue.cc:1035] T 8e5a31a539924474b31545a7c6737425 P f7f31863969a4bda9e128166c15ac1ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "3176a36c35d2480fb87fa57226b3bd7d" member_type: VOTER last_known_addr { host: "127.1.170.131" port: 45757 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 20:09:51.886062 9473 raft_consensus.cc:1261] T 8e5a31a539924474b31545a7c6737425 P c8c21250d251462898a337ffe611517e [term 1 FOLLOWER]: Refusing update from remote peer f7f31863969a4bda9e128166c15ac1ad: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 20:09:51.886781 9926 consensus_queue.cc:1035] T 8e5a31a539924474b31545a7c6737425 P f7f31863969a4bda9e128166c15ac1ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8c21250d251462898a337ffe611517e" member_type: VOTER last_known_addr { host: "127.1.170.129" port: 34387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 20:09:51.952881 9314 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50442:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 20:09:51.953157 9314 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50442:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 20:09:51.956212 9314 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6b22557909954a558ccd24b8fbbc4f78: Sending DeleteTablet for 3 replicas of tablet 8e5a31a539924474b31545a7c6737425
I20260923 20:09:51.957007 9591 tablet_service.cc:1514] Processing DeleteTablet for tablet 8e5a31a539924474b31545a7c6737425 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 20:09:51 UTC) from {username='oryx', principal='oryx/127.1.170.190@KRBTEST.COM'} at 127.0.0.1:57368
I20260923 20:09:51.957012 9452 tablet_service.cc:1514] Processing DeleteTablet for tablet 8e5a31a539924474b31545a7c6737425 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 20:09:51 UTC) from {username='oryx', principal='oryx/127.1.170.190@KRBTEST.COM'} at 127.0.0.1:47884
I20260923 20:09:51.957305 9729 tablet_service.cc:1514] Processing DeleteTablet for tablet 8e5a31a539924474b31545a7c6737425 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 20:09:51 UTC) from {username='oryx', principal='oryx/127.1.170.190@KRBTEST.COM'} at 127.0.0.1:59158
I20260923 20:09:51.957614 9890 tablet_replica.cc:330] T 8e5a31a539924474b31545a7c6737425 P f7f31863969a4bda9e128166c15ac1ad: stopping tablet replica
I20260923 20:09:51.957643 9891 tablet_replica.cc:330] T 8e5a31a539924474b31545a7c6737425 P 3176a36c35d2480fb87fa57226b3bd7d: stopping tablet replica
I20260923 20:09:51.957773 9890 raft_consensus.cc:2229] T 8e5a31a539924474b31545a7c6737425 P f7f31863969a4bda9e128166c15ac1ad [term 1 LEADER]: Raft consensus shutting down.
I20260923 20:09:51.957815 9891 raft_consensus.cc:2229] T 8e5a31a539924474b31545a7c6737425 P 3176a36c35d2480fb87fa57226b3bd7d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 20:09:51.958047 9891 raft_consensus.cc:2258] T 8e5a31a539924474b31545a7c6737425 P 3176a36c35d2480fb87fa57226b3bd7d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 20:09:51.958091 9890 raft_consensus.cc:2258] T 8e5a31a539924474b31545a7c6737425 P f7f31863969a4bda9e128166c15ac1ad [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 20:09:51.958395 9892 tablet_replica.cc:330] T 8e5a31a539924474b31545a7c6737425 P c8c21250d251462898a337ffe611517e: stopping tablet replica
I20260923 20:09:51.958598 9892 raft_consensus.cc:2229] T 8e5a31a539924474b31545a7c6737425 P c8c21250d251462898a337ffe611517e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 20:09:51.958828 9892 raft_consensus.cc:2258] T 8e5a31a539924474b31545a7c6737425 P c8c21250d251462898a337ffe611517e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 20:09:51.959175 9891 ts_tablet_manager.cc:1905] T 8e5a31a539924474b31545a7c6737425 P 3176a36c35d2480fb87fa57226b3bd7d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 20:09:51.960104 9890 ts_tablet_manager.cc:1905] T 8e5a31a539924474b31545a7c6737425 P f7f31863969a4bda9e128166c15ac1ad: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 20:09:51.960111 9892 ts_tablet_manager.cc:1905] T 8e5a31a539924474b31545a7c6737425 P c8c21250d251462898a337ffe611517e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 20:09:51.965924 9891 ts_tablet_manager.cc:1918] T 8e5a31a539924474b31545a7c6737425 P 3176a36c35d2480fb87fa57226b3bd7d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260923 20:09:51.966082 9891 log.cc:1196] T 8e5a31a539924474b31545a7c6737425 P 3176a36c35d2480fb87fa57226b3bd7d: Deleting WAL directory at /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790194121577027-1706-0/minicluster-data/ts-2/wal/wals/8e5a31a539924474b31545a7c6737425
I20260923 20:09:51.966526 9891 ts_tablet_manager.cc:1939] T 8e5a31a539924474b31545a7c6737425 P 3176a36c35d2480fb87fa57226b3bd7d: Deleting consensus metadata
I20260923 20:09:51.966982 1706 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskqUuku6/build/asan/bin/kudu pid 9386
I20260923 20:09:51.967188 9890 ts_tablet_manager.cc:1918] T 8e5a31a539924474b31545a7c6737425 P f7f31863969a4bda9e128166c15ac1ad: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260923 20:09:51.967351 9890 log.cc:1196] T 8e5a31a539924474b31545a7c6737425 P f7f31863969a4bda9e128166c15ac1ad: Deleting WAL directory at /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790194121577027-1706-0/minicluster-data/ts-1/wal/wals/8e5a31a539924474b31545a7c6737425
I20260923 20:09:51.967713 9301 catalog_manager.cc:4841] TS 3176a36c35d2480fb87fa57226b3bd7d (127.1.170.131:45757): tablet 8e5a31a539924474b31545a7c6737425 (table test-table [id=51f3999106704c69b05b1b8a8b019a3b]) successfully deleted
I20260923 20:09:51.967785 9890 ts_tablet_manager.cc:1939] T 8e5a31a539924474b31545a7c6737425 P f7f31863969a4bda9e128166c15ac1ad: Deleting consensus metadata
I20260923 20:09:51.969604 9299 catalog_manager.cc:4841] TS f7f31863969a4bda9e128166c15ac1ad (127.1.170.130:35035): tablet 8e5a31a539924474b31545a7c6737425 (table test-table [id=51f3999106704c69b05b1b8a8b019a3b]) successfully deleted
I20260923 20:09:51.971668 9892 ts_tablet_manager.cc:1918] T 8e5a31a539924474b31545a7c6737425 P c8c21250d251462898a337ffe611517e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260923 20:09:51.971866 9892 log.cc:1196] T 8e5a31a539924474b31545a7c6737425 P c8c21250d251462898a337ffe611517e: Deleting WAL directory at /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790194121577027-1706-0/minicluster-data/ts-0/wal/wals/8e5a31a539924474b31545a7c6737425
I20260923 20:09:51.972360 9892 ts_tablet_manager.cc:1939] T 8e5a31a539924474b31545a7c6737425 P c8c21250d251462898a337ffe611517e: Deleting consensus metadata
I20260923 20:09:51.973418 9299 catalog_manager.cc:4841] TS c8c21250d251462898a337ffe611517e (127.1.170.129:34387): tablet 8e5a31a539924474b31545a7c6737425 (table test-table [id=51f3999106704c69b05b1b8a8b019a3b]) successfully deleted
Sep 23 20:09:51 dist-test-slave-th4g krb5kdc[9270](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194191, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.1.170.129@KRBTEST.COM
I20260923 20:09:51.985787 9913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:09:51.968479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47908 (local address 127.1.170.129:34387)
0923 20:09:51.968665 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0923 20:09:51.968671 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:09:51.968694 (+ 23us) server_negotiation.cc:407] Connection header received
0923 20:09:51.968915 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:09:51.968923 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:09:51.969007 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:09:51.969093 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:09:51.970905 (+ 1812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:09:51.972083 (+ 1178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:09:51.973817 (+ 1734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:09:51.974534 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:09:51.974656 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:09:51.978668 (+ 4012us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:09:51.978681 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:09:51.978699 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:09:51.978738 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:09:51.982856 (+ 4118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:09:51.983551 (+ 695us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:09:51.983560 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:09:51.983566 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:09:51.983736 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:09:51.984136 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:09:51.984141 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:09:51.984144 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:09:51.984594 (+ 450us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:09:51.984906 (+ 312us) server_negotiation.cc:1035] Waiting for connection context
0923 20:09:51.985134 (+ 228us) server_negotiation.cc:299] Negotiation successful
0923 20:09:51.985396 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260923 20:09:51.986624 9517 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:09:52.113736 1706 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqUuku6/build/asan/bin/kudu with pid 9386
I20260923 20:09:52.135257 1706 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskqUuku6/build/asan/bin/kudu pid 9526
Sep 23 20:09:52 dist-test-slave-th4g krb5kdc[9270](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194191, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.1.170.130@KRBTEST.COM
I20260923 20:09:52.149631 9824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:09:52.135964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57402 (local address 127.1.170.130:35035)
0923 20:09:52.136131 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0923 20:09:52.136135 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 20:09:52.136189 (+ 54us) server_negotiation.cc:407] Connection header received
0923 20:09:52.136339 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:09:52.136343 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:09:52.136437 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:09:52.136521 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:09:52.137741 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:09:52.138501 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:09:52.139785 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:09:52.140267 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:09:52.140373 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:09:52.144592 (+ 4219us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:09:52.144603 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:09:52.144619 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:09:52.144650 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:09:52.147324 (+ 2674us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:09:52.147952 (+ 628us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:09:52.147958 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:09:52.147961 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:09:52.148079 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:09:52.148406 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:09:52.148409 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:09:52.148411 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:09:52.148741 (+ 330us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:09:52.148962 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0923 20:09:52.149198 (+ 236us) server_negotiation.cc:299] Negotiation successful
0923 20:09:52.149375 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260923 20:09:52.150400 9653 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:09:52.294704 1706 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqUuku6/build/asan/bin/kudu with pid 9526
I20260923 20:09:52.321098 1706 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskqUuku6/build/asan/bin/kudu pid 9665
Sep 23 20:09:52 dist-test-slave-th4g krb5kdc[9270](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194191, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.1.170.131@KRBTEST.COM
I20260923 20:09:52.343279 9939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:09:52.321870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59168 (local address 127.1.170.131:45757)
0923 20:09:52.322437 (+ 567us) server_negotiation.cc:206] Beginning negotiation
0923 20:09:52.322441 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 20:09:52.322455 (+ 14us) server_negotiation.cc:407] Connection header received
0923 20:09:52.322505 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:09:52.322508 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:09:52.322568 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:09:52.322694 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:09:52.323923 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:09:52.324601 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:09:52.325803 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:09:52.326261 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:09:52.326398 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:09:52.336961 (+ 10563us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:09:52.336978 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:09:52.336997 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:09:52.337056 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:09:52.340092 (+ 3036us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:09:52.340896 (+ 804us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:09:52.340903 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:09:52.340906 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:09:52.341045 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:09:52.341415 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:09:52.341422 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:09:52.341425 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:09:52.341997 (+ 572us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:09:52.342301 (+ 304us) server_negotiation.cc:1035] Waiting for connection context
0923 20:09:52.342652 (+ 351us) server_negotiation.cc:299] Negotiation successful
0923 20:09:52.342939 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":399,"thread_start_us":217,"threads_started":1}
I20260923 20:09:52.344015 9795 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:09:52.467024 1706 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqUuku6/build/asan/bin/kudu with pid 9665
I20260923 20:09:52.488867 1706 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskqUuku6/build/asan/bin/kudu pid 9283
I20260923 20:09:52.490154 9355 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:09:52.606254 1706 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqUuku6/build/asan/bin/kudu with pid 9283
2026-09-23T20:09:52Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (7251 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790194121577027-1706-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 20:09:52 dist-test-slave-th4g krb5kdc[9944](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 20:09:52 dist-test-slave-th4g krb5kdc[9944](info): set up 2 sockets
Sep 23 20:09:52 dist-test-slave-th4g krb5kdc[9944](info): commencing operation
krb5kdc: starting...
W20260923 20:09:53.757061 1706 mini_kdc.cc:121] Time spent starting KDC: real 1.106s user 0.000s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790194121577027-1706-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790194121577027-1706-0/krb5kdc/test-user.keytab.
Sep 23 20:09:53 dist-test-slave-th4g krb5kdc[9944](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194193, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T20:09:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T20:09:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.170.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.170.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.170.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790194121577027-1706-0/krb5kdc/kudu_127.1.170.190.keytab.
Entry for principal kudu/127.1.170.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790194121577027-1706-0/krb5kdc/kudu_127.1.170.190.keytab.
I20260923 20:09:53.998579 1706 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqUuku6/build/asan/bin/kudu
/tmp/dist-test-taskqUuku6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790194121577027-1706-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790194121577027-1706-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790194121577027-1706-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790194121577027-1706-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.170.190:42711
--webserver_interface=127.1.170.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790194121577027-1706-0/krb5kdc/kudu_127.1.170.190.keytab
--principal=kudu/127.1.170.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.170.148:43763
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.170.190:42711
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790194121577027-1706-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790194121577027-1706-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790194121577027-1706-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:09:54.343797 9957 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:09:54.344290 9957 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:09:54.344476 9957 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:09:54.352048 9957 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 20:09:54.352243 9957 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:09:54.352361 9957 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:09:54.352471 9957 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 20:09:54.352568 9957 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 20:09:54.362634 9957 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.170.148:43763
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790194121577027-1706-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790194121577027-1706-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.170.190:42711
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790194121577027-1706-0/krb5kdc/kudu_127.1.170.190.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.170.190:42711
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790194121577027-1706-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.170.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790194121577027-1706-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:09:54.364055 9957 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:09:54.365406 9957 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:09:54.378459 9962 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:09:54.378423 9965 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:09:54.378587 9963 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:09:54.384397 9957 server_base.cc:1034] running on GCE node
I20260923 20:09:54.385751 9957 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:09:54.388168 9957 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:09:54.389544 9957 hybrid_clock.cc:648] HybridClock initialized: now 1790194194389437 us; error 74 us; skew 500 ppm
I20260923 20:09:54.390151 9957 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-09-23T20:09:54Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1853 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194121577027-1706-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 20:09:54 dist-test-slave-th4g krb5kdc[9971](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 20:09:54 dist-test-slave-th4g krb5kdc[9971](info): set up 2 sockets
Sep 23 20:09:54 dist-test-slave-th4g krb5kdc[9971](info): commencing operation
krb5kdc: starting...
W20260923 20:09:55.596125 1706 mini_kdc.cc:121] Time spent starting KDC: real 1.092s user 0.000s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194121577027-1706-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194121577027-1706-0/krb5kdc/test-user.keytab.
Sep 23 20:09:55 dist-test-slave-th4g krb5kdc[9971](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194195, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T20:09:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T20:09:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.170.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.170.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.170.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194121577027-1706-0/krb5kdc/kudu_127.1.170.190.keytab.
Entry for principal kudu/127.1.170.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194121577027-1706-0/krb5kdc/kudu_127.1.170.190.keytab.
I20260923 20:09:55.826637 1706 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqUuku6/build/asan/bin/kudu
/tmp/dist-test-taskqUuku6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194121577027-1706-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194121577027-1706-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194121577027-1706-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194121577027-1706-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.170.190:36595
--webserver_interface=127.1.170.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194121577027-1706-0/krb5kdc/kudu_127.1.170.190.keytab
--principal=kudu/127.1.170.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.170.148:38359
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.170.190:36595
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194121577027-1706-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194121577027-1706-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194121577027-1706-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:09:56.152957 9984 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:09:56.153421 9984 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:09:56.153661 9984 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:09:56.162415 9984 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 20:09:56.162550 9984 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:09:56.162644 9984 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:09:56.162748 9984 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 20:09:56.162830 9984 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 20:09:56.173086 9984 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.170.148:38359
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194121577027-1706-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194121577027-1706-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.170.190:36595
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194121577027-1706-0/krb5kdc/kudu_127.1.170.190.keytab
--principal=kudu/127.1.170.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.170.190:36595
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194121577027-1706-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.170.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194121577027-1706-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:09:56.174566 9984 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:09:56.175881 9984 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:09:56.187928 9992 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:09:56.188063 9989 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:09:56.187973 9990 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:09:56.189145 9984 server_base.cc:1034] running on GCE node
I20260923 20:09:56.190038 9984 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:09:56.191514 9984 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:09:56.192714 9984 hybrid_clock.cc:648] HybridClock initialized: now 1790194196192661 us; error 56 us; skew 500 ppm
I20260923 20:09:56.193060 9984 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:09:56 dist-test-slave-th4g krb5kdc[9971](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194196, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.170.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:09:56.196334 9984 init.cc:377] Logged in from keytab as kudu/127.1.170.190@KRBTEST.COM (short username kudu)
I20260923 20:09:56.197918 9984 webserver.cc:460] Webserver started at http://127.1.170.190:41157/ using document root <none> and password file <none>
I20260923 20:09:56.198591 9984 fs_manager.cc:359] Metadata directory not provided
I20260923 20:09:56.198688 9984 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:09:56.198936 9984 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:09:56.202078 9984 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194121577027-1706-0/minicluster-data/master-0/data/instance:
uuid: "40fdd4262e6c4efa855ec33d169d0953"
format_stamp: "Formatted at 2026-09-23 20:09:56 on dist-test-slave-th4g"
server_key: "71ce9fa5a34c53992a56b3f6ac69b42f"
server_key_iv: "22b4be7949884a5dbc210bfba42acc40"
server_key_version: "encryptionkey@0"
I20260923 20:09:56.202812 9984 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194121577027-1706-0/minicluster-data/master-0/wal/instance:
uuid: "40fdd4262e6c4efa855ec33d169d0953"
format_stamp: "Formatted at 2026-09-23 20:09:56 on dist-test-slave-th4g"
server_key: "71ce9fa5a34c53992a56b3f6ac69b42f"
server_key_iv: "22b4be7949884a5dbc210bfba42acc40"
server_key_version: "encryptionkey@0"
I20260923 20:09:56.207975 9984 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.002s
I20260923 20:09:56.211382 9999 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:09:56.212713 9984 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260923 20:09:56.212908 9984 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194121577027-1706-0/minicluster-data/master-0/data,/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194121577027-1706-0/minicluster-data/master-0/wal
uuid: "40fdd4262e6c4efa855ec33d169d0953"
format_stamp: "Formatted at 2026-09-23 20:09:56 on dist-test-slave-th4g"
server_key: "71ce9fa5a34c53992a56b3f6ac69b42f"
server_key_iv: "22b4be7949884a5dbc210bfba42acc40"
server_key_version: "encryptionkey@0"
I20260923 20:09:56.213129 9984 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194121577027-1706-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194121577027-1706-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194121577027-1706-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:09:56.226536 9984 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:09:56.230440 9984 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:09:56.230728 9984 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:09:56.250010 9984 rpc_server.cc:307] RPC server started. Bound to: 127.1.170.190:36595
I20260923 20:09:56.250048 10050 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.170.190:36595 every 8 connection(s)
I20260923 20:09:56.251431 9984 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194121577027-1706-0/minicluster-data/master-0/data/info.pb
I20260923 20:09:56.255339 10051 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:09:56.259074 1706 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqUuku6/build/asan/bin/kudu as pid 9984
I20260923 20:09:56.259325 1706 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194121577027-1706-0/minicluster-data/master-0/wal/instance
I20260923 20:09:56.259948 1706 external_mini_cluster.cc:1428] Setting key 5be4b58f896679b3007c99dc86439e05
I20260923 20:09:56.268009 10051 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 40fdd4262e6c4efa855ec33d169d0953: Bootstrap starting.
I20260923 20:09:56.271802 10051 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 40fdd4262e6c4efa855ec33d169d0953: Neither blocks nor log segments found. Creating new log.
I20260923 20:09:56.273124 10051 log.cc:824] T 00000000000000000000000000000000 P 40fdd4262e6c4efa855ec33d169d0953: Log is configured to *not* fsync() on all Append() calls
I20260923 20:09:56.276201 10051 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 40fdd4262e6c4efa855ec33d169d0953: No bootstrap required, opened a new log
Sep 23 20:09:56 dist-test-slave-th4g krb5kdc[9971](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194195, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.170.190@KRBTEST.COM
I20260923 20:09:56.286441 10054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:09:56.262043 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39464 (local address 127.1.170.190:36595)
0923 20:09:56.262866 (+ 823us) server_negotiation.cc:206] Beginning negotiation
0923 20:09:56.262880 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0923 20:09:56.262933 (+ 53us) server_negotiation.cc:407] Connection header received
0923 20:09:56.266194 (+ 3261us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:09:56.266226 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:09:56.266774 (+ 548us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:09:56.267249 (+ 475us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:09:56.268698 (+ 1449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:09:56.270833 (+ 2135us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:09:56.272224 (+ 1391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:09:56.273318 (+ 1094us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:09:56.273488 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:09:56.277499 (+ 4011us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:09:56.277512 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:09:56.277531 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:09:56.277584 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:09:56.281605 (+ 4021us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:09:56.282425 (+ 820us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:09:56.282435 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:09:56.282443 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:09:56.282668 (+ 225us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:09:56.283095 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:09:56.283099 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:09:56.283102 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:09:56.283845 (+ 743us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:09:56.284153 (+ 308us) server_negotiation.cc:1035] Waiting for connection context
0923 20:09:56.284486 (+ 333us) server_negotiation.cc:299] Negotiation successful
0923 20:09:56.284957 (+ 471us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":580,"thread_start_us":359,"threads_started":1}
I20260923 20:09:56.287322 10051 raft_consensus.cc:348] T 00000000000000000000000000000000 P 40fdd4262e6c4efa855ec33d169d0953 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40fdd4262e6c4efa855ec33d169d0953" member_type: VOTER last_known_addr { host: "127.1.170.190" port: 36595 } }
I20260923 20:09:56.287738 10051 raft_consensus.cc:374] T 00000000000000000000000000000000 P 40fdd4262e6c4efa855ec33d169d0953 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:09:56.287887 10051 raft_consensus.cc:729] T 00000000000000000000000000000000 P 40fdd4262e6c4efa855ec33d169d0953 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 40fdd4262e6c4efa855ec33d169d0953, State: Initialized, Role: FOLLOWER
I20260923 20:09:56.288504 10051 consensus_queue.cc:260] T 00000000000000000000000000000000 P 40fdd4262e6c4efa855ec33d169d0953 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40fdd4262e6c4efa855ec33d169d0953" member_type: VOTER last_known_addr { host: "127.1.170.190" port: 36595 } }
I20260923 20:09:56.288766 10051 raft_consensus.cc:388] T 00000000000000000000000000000000 P 40fdd4262e6c4efa855ec33d169d0953 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 20:09:56.288925 10051 raft_consensus.cc:482] T 00000000000000000000000000000000 P 40fdd4262e6c4efa855ec33d169d0953 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 20:09:56.289106 10051 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 40fdd4262e6c4efa855ec33d169d0953 [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:09:56.291432 10051 raft_consensus.cc:504] T 00000000000000000000000000000000 P 40fdd4262e6c4efa855ec33d169d0953 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40fdd4262e6c4efa855ec33d169d0953" member_type: VOTER last_known_addr { host: "127.1.170.190" port: 36595 } }
I20260923 20:09:56.291920 10051 leader_election.cc:302] T 00000000000000000000000000000000 P 40fdd4262e6c4efa855ec33d169d0953 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 40fdd4262e6c4efa855ec33d169d0953; no voters:
I20260923 20:09:56.292508 10051 leader_election.cc:288] T 00000000000000000000000000000000 P 40fdd4262e6c4efa855ec33d169d0953 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 20:09:56.292658 10056 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 40fdd4262e6c4efa855ec33d169d0953 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 20:09:56.293399 10056 raft_consensus.cc:686] T 00000000000000000000000000000000 P 40fdd4262e6c4efa855ec33d169d0953 [term 1 LEADER]: Becoming Leader. State: Replica: 40fdd4262e6c4efa855ec33d169d0953, State: Running, Role: LEADER
I20260923 20:09:56.294026 10056 consensus_queue.cc:237] T 00000000000000000000000000000000 P 40fdd4262e6c4efa855ec33d169d0953 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40fdd4262e6c4efa855ec33d169d0953" member_type: VOTER last_known_addr { host: "127.1.170.190" port: 36595 } }
I20260923 20:09:56.294736 10051 sys_catalog.cc:564] T 00000000000000000000000000000000 P 40fdd4262e6c4efa855ec33d169d0953 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 20:09:56.296969 10057 sys_catalog.cc:455] T 00000000000000000000000000000000 P 40fdd4262e6c4efa855ec33d169d0953 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "40fdd4262e6c4efa855ec33d169d0953" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40fdd4262e6c4efa855ec33d169d0953" member_type: VOTER last_known_addr { host: "127.1.170.190" port: 36595 } } }
I20260923 20:09:56.297456 10057 sys_catalog.cc:458] T 00000000000000000000000000000000 P 40fdd4262e6c4efa855ec33d169d0953 [sys.catalog]: This master's current role is: LEADER
I20260923 20:09:56.297084 10058 sys_catalog.cc:455] T 00000000000000000000000000000000 P 40fdd4262e6c4efa855ec33d169d0953 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 40fdd4262e6c4efa855ec33d169d0953. Latest consensus state: current_term: 1 leader_uuid: "40fdd4262e6c4efa855ec33d169d0953" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40fdd4262e6c4efa855ec33d169d0953" member_type: VOTER last_known_addr { host: "127.1.170.190" port: 36595 } } }
I20260923 20:09:56.299822 10058 sys_catalog.cc:458] T 00000000000000000000000000000000 P 40fdd4262e6c4efa855ec33d169d0953 [sys.catalog]: This master's current role is: LEADER
I20260923 20:09:56.301393 10065 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 20:09:56.306902 10065 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 20:09:56.316221 10065 catalog_manager.cc:1347] Generated new cluster ID: 765d13068e3e4084a16f3875bac23574
I20260923 20:09:56.316413 10065 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 20:09:56.331228 10065 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 20:09:56.332656 10065 catalog_manager.cc:1504] Loading token signing keys...
I20260923 20:09:56.342909 10065 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 40fdd4262e6c4efa855ec33d169d0953: Generated new TSK 0
I20260923 20:09:56.343828 10065 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.170.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.170.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.170.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194121577027-1706-0/krb5kdc/kudu_127.1.170.129.keytab.
Entry for principal kudu/127.1.170.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194121577027-1706-0/krb5kdc/kudu_127.1.170.129.keytab.
I20260923 20:09:56.406035 1706 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqUuku6/build/asan/bin/kudu
/tmp/dist-test-taskqUuku6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194121577027-1706-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194121577027-1706-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194121577027-1706-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194121577027-1706-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.170.129:0
--local_ip_for_outbound_sockets=127.1.170.129
--webserver_interface=127.1.170.129
--webserver_port=0
--tserver_master_addrs=127.1.170.190:36595
--keytab_file=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194121577027-1706-0/krb5kdc/kudu_127.1.170.129.keytab
--principal=kudu/127.1.170.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.170.148:38359
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194121577027-1706-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194121577027-1706-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194121577027-1706-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:09:56.736264 10077 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:09:56.736599 10077 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:09:56.736788 10077 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:09:56.744127 10077 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:09:56.744302 10077 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:09:56.744490 10077 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.170.129
I20260923 20:09:56.754416 10077 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.170.148:38359
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194121577027-1706-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194121577027-1706-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194121577027-1706-0/krb5kdc/kudu_127.1.170.129.keytab
--principal=kudu/127.1.170.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.170.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194121577027-1706-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.170.129
--webserver_port=0
--tserver_master_addrs=127.1.170.190:36595
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.170.129
--log_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194121577027-1706-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:09:56.755818 10077 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:09:56.757072 10077 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:09:56.770040 10083 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:09:56.770001 10085 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:09:56.770437 10082 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:09:56.770637 10077 server_base.cc:1034] running on GCE node
I20260923 20:09:56.771306 10077 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:09:56.772233 10077 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:09:56.773443 10077 hybrid_clock.cc:648] HybridClock initialized: now 1790194196773385 us; error 52 us; skew 500 ppm
I20260923 20:09:56.773777 10077 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:09:56 dist-test-slave-th4g krb5kdc[9971](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194196, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.170.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:09:56.776798 10077 init.cc:377] Logged in from keytab as kudu/127.1.170.129@KRBTEST.COM (short username kudu)
I20260923 20:09:56.778139 10077 webserver.cc:460] Webserver started at http://127.1.170.129:41889/ using document root <none> and password file <none>
I20260923 20:09:56.778756 10077 fs_manager.cc:359] Metadata directory not provided
I20260923 20:09:56.778848 10077 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:09:56.779084 10077 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:09:56.781777 10077 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194121577027-1706-0/minicluster-data/ts-0/data/instance:
uuid: "0bfbaf6365cf4baea93de6401ebcc278"
format_stamp: "Formatted at 2026-09-23 20:09:56 on dist-test-slave-th4g"
server_key: "3c1fdc0fae70f1ca246d8297da00edbc"
server_key_iv: "cad483cd70c9dba3d3a06c11c915ccf1"
server_key_version: "encryptionkey@0"
I20260923 20:09:56.782534 10077 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194121577027-1706-0/minicluster-data/ts-0/wal/instance:
uuid: "0bfbaf6365cf4baea93de6401ebcc278"
format_stamp: "Formatted at 2026-09-23 20:09:56 on dist-test-slave-th4g"
server_key: "3c1fdc0fae70f1ca246d8297da00edbc"
server_key_iv: "cad483cd70c9dba3d3a06c11c915ccf1"
server_key_version: "encryptionkey@0"
I20260923 20:09:56.787328 10077 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260923 20:09:56.790624 10092 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:09:56.791919 10077 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260923 20:09:56.792050 10077 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194121577027-1706-0/minicluster-data/ts-0/data,/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194121577027-1706-0/minicluster-data/ts-0/wal
uuid: "0bfbaf6365cf4baea93de6401ebcc278"
format_stamp: "Formatted at 2026-09-23 20:09:56 on dist-test-slave-th4g"
server_key: "3c1fdc0fae70f1ca246d8297da00edbc"
server_key_iv: "cad483cd70c9dba3d3a06c11c915ccf1"
server_key_version: "encryptionkey@0"
I20260923 20:09:56.792279 10077 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194121577027-1706-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194121577027-1706-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194121577027-1706-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:09:56.805042 10077 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:09:56.809008 10077 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:09:56.809295 10077 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:09:56.810245 10077 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 20:09:56.811506 10077 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 20:09:56.811578 10077 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 20:09:56.811726 10077 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 20:09:56.811808 10077 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 20:09:56.845435 10077 rpc_server.cc:307] RPC server started. Bound to: 127.1.170.129:44063
I20260923 20:09:56.845480 10204 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.170.129:44063 every 8 connection(s)
I20260923 20:09:56.846760 10077 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194121577027-1706-0/minicluster-data/ts-0/data/info.pb
I20260923 20:09:56.850980 1706 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqUuku6/build/asan/bin/kudu as pid 10077
I20260923 20:09:56.851202 1706 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194121577027-1706-0/minicluster-data/ts-0/wal/instance
I20260923 20:09:56.851991 1706 external_mini_cluster.cc:1428] Setting key 1635f625845adbe00e47a8bdf02ac796
Sep 23 20:09:56 dist-test-slave-th4g krb5kdc[9971](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194196, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.170.129@KRBTEST.COM for kudu/127.1.170.190@KRBTEST.COM
I20260923 20:09:56.871358 10207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:09:56.849183 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.170.129:58251 (local address 127.1.170.190:36595)
0923 20:09:56.849636 (+ 453us) server_negotiation.cc:206] Beginning negotiation
0923 20:09:56.849640 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 20:09:56.850412 (+ 772us) server_negotiation.cc:407] Connection header received
0923 20:09:56.853259 (+ 2847us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:09:56.853265 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:09:56.853338 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:09:56.853462 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:09:56.855569 (+ 2107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:09:56.856402 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:09:56.858984 (+ 2582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:09:56.859425 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:09:56.859533 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:09:56.864145 (+ 4612us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:09:56.864156 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:09:56.864164 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:09:56.864214 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:09:56.867789 (+ 3575us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:09:56.868636 (+ 847us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:09:56.868646 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:09:56.868652 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:09:56.868809 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:09:56.869239 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:09:56.869246 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:09:56.869252 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:09:56.869654 (+ 402us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:09:56.869833 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0923 20:09:56.870610 (+ 777us) server_negotiation.cc:299] Negotiation successful
0923 20:09:56.870881 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":344,"thread_start_us":256,"threads_started":1}
I20260923 20:09:56.872041 10208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:09:56.849527 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.190:36595 (local address 127.1.170.129:58251)
0923 20:09:56.850232 (+ 705us) negotiation.cc:107] Waiting for socket to connect
0923 20:09:56.850268 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0923 20:09:56.852743 (+ 2475us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:09:56.853702 (+ 959us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:09:56.853710 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:09:56.854499 (+ 789us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:09:56.855339 (+ 840us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:09:56.855366 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:09:56.856593 (+ 1227us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:09:56.856602 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:09:56.858782 (+ 2180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:09:56.858807 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:09:56.859674 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:09:56.859683 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:09:56.860039 (+ 356us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:09:56.860901 (+ 862us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:09:56.860936 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:09:56.863834 (+ 2898us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:09:56.867994 (+ 4160us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:09:56.868003 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:09:56.868018 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:09:56.868484 (+ 466us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:09:56.868929 (+ 445us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:09:56.868932 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:09:56.868935 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:09:56.869120 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:09:56.869785 (+ 665us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:09:56.869791 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:09:56.870201 (+ 410us) client_negotiation.cc:769] Sending connection context
0923 20:09:56.870445 (+ 244us) client_negotiation.cc:240] Negotiation successful
0923 20:09:56.870864 (+ 419us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":506,"thread_start_us":315,"threads_started":1}
I20260923 20:09:56.873749 10205 heartbeater.cc:347] Connected to a master server at 127.1.170.190:36595
I20260923 20:09:56.874091 10205 heartbeater.cc:464] Registering TS with master...
I20260923 20:09:56.874847 10205 heartbeater.cc:511] Master 127.1.170.190:36595 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.1.170.130@KRBTEST.COM; defaulting to no policy
I20260923 20:09:56.877036 10016 ts_manager.cc:194] Registered new tserver with Master: 0bfbaf6365cf4baea93de6401ebcc278 (127.1.170.129:44063)
I20260923 20:09:56.878648 10016 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.170.129@KRBTEST.COM'} at 127.1.170.129:58251
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.170.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.170.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194121577027-1706-0/krb5kdc/kudu_127.1.170.130.keytab.
Entry for principal kudu/127.1.170.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194121577027-1706-0/krb5kdc/kudu_127.1.170.130.keytab.
I20260923 20:09:56.910327 1706 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqUuku6/build/asan/bin/kudu
/tmp/dist-test-taskqUuku6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194121577027-1706-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194121577027-1706-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194121577027-1706-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194121577027-1706-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.170.130:0
--local_ip_for_outbound_sockets=127.1.170.130
--webserver_interface=127.1.170.130
--webserver_port=0
--tserver_master_addrs=127.1.170.190:36595
--keytab_file=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194121577027-1706-0/krb5kdc/kudu_127.1.170.130.keytab
--principal=kudu/127.1.170.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.170.148:38359
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194121577027-1706-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194121577027-1706-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194121577027-1706-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:09:57.227636 10211 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:09:57.227922 10211 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:09:57.228091 10211 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:09:57.235450 10211 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:09:57.235575 10211 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:09:57.235761 10211 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.170.130
I20260923 20:09:57.245529 10211 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.170.148:38359
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194121577027-1706-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194121577027-1706-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194121577027-1706-0/krb5kdc/kudu_127.1.170.130.keytab
--principal=kudu/127.1.170.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.170.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194121577027-1706-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.170.130
--webserver_port=0
--tserver_master_addrs=127.1.170.190:36595
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.170.130
--log_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194121577027-1706-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:09:57.246848 10211 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:09:57.248100 10211 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:09:57.263054 10216 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:09:57.263293 10219 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:09:57.263458 10217 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:09:57.263971 10211 server_base.cc:1034] running on GCE node
I20260923 20:09:57.264658 10211 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:09:57.265750 10211 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:09:57.266995 10211 hybrid_clock.cc:648] HybridClock initialized: now 1790194197266914 us; error 62 us; skew 500 ppm
I20260923 20:09:57.267336 10211 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:09:57 dist-test-slave-th4g krb5kdc[9971](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194197, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.170.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:09:57.270926 10211 init.cc:377] Logged in from keytab as kudu/127.1.170.130@KRBTEST.COM (short username kudu)
I20260923 20:09:57.272334 10211 webserver.cc:460] Webserver started at http://127.1.170.130:37077/ using document root <none> and password file <none>
I20260923 20:09:57.273023 10211 fs_manager.cc:359] Metadata directory not provided
I20260923 20:09:57.273175 10211 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:09:57.273504 10211 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:09:57.276373 10211 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194121577027-1706-0/minicluster-data/ts-1/data/instance:
uuid: "0790313c2d224e8fb772982e48a057ab"
format_stamp: "Formatted at 2026-09-23 20:09:57 on dist-test-slave-th4g"
server_key: "540df7a8190cc9cc4d3533f2d0747bf9"
server_key_iv: "e00c51e18cd99695b021a3166361d0e8"
server_key_version: "encryptionkey@0"
I20260923 20:09:57.277143 10211 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194121577027-1706-0/minicluster-data/ts-1/wal/instance:
uuid: "0790313c2d224e8fb772982e48a057ab"
format_stamp: "Formatted at 2026-09-23 20:09:57 on dist-test-slave-th4g"
server_key: "540df7a8190cc9cc4d3533f2d0747bf9"
server_key_iv: "e00c51e18cd99695b021a3166361d0e8"
server_key_version: "encryptionkey@0"
I20260923 20:09:57.282366 10211 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260923 20:09:57.285802 10226 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:09:57.287297 10211 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260923 20:09:57.287475 10211 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194121577027-1706-0/minicluster-data/ts-1/data,/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194121577027-1706-0/minicluster-data/ts-1/wal
uuid: "0790313c2d224e8fb772982e48a057ab"
format_stamp: "Formatted at 2026-09-23 20:09:57 on dist-test-slave-th4g"
server_key: "540df7a8190cc9cc4d3533f2d0747bf9"
server_key_iv: "e00c51e18cd99695b021a3166361d0e8"
server_key_version: "encryptionkey@0"
I20260923 20:09:57.287669 10211 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194121577027-1706-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194121577027-1706-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194121577027-1706-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:09:57.314157 10211 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:09:57.319276 10211 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:09:57.319588 10211 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:09:57.320787 10211 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 20:09:57.322331 10211 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 20:09:57.322407 10211 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 20:09:57.322540 10211 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 20:09:57.322597 10211 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 20:09:57.356663 10211 rpc_server.cc:307] RPC server started. Bound to: 127.1.170.130:45381
I20260923 20:09:57.356745 10338 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.170.130:45381 every 8 connection(s)
I20260923 20:09:57.357941 10211 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194121577027-1706-0/minicluster-data/ts-1/data/info.pb
I20260923 20:09:57.362249 1706 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqUuku6/build/asan/bin/kudu as pid 10211
I20260923 20:09:57.362434 1706 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194121577027-1706-0/minicluster-data/ts-1/wal/instance
I20260923 20:09:57.362838 1706 external_mini_cluster.cc:1428] Setting key 7e27dd823326e3e6671f19d8fa5e51d3
Sep 23 20:09:57 dist-test-slave-th4g krb5kdc[9971](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194197, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.170.130@KRBTEST.COM for kudu/127.1.170.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.1.170.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.170.131@KRBTEST.COM" created.
I20260923 20:09:57.387071 10207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:09:57.360614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.170.130:58025 (local address 127.1.170.190:36595)
0923 20:09:57.360835 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0923 20:09:57.360840 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 20:09:57.361855 (+ 1015us) server_negotiation.cc:407] Connection header received
0923 20:09:57.363982 (+ 2127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:09:57.363987 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:09:57.364075 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:09:57.364185 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:09:57.365707 (+ 1522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:09:57.366484 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:09:57.368013 (+ 1529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:09:57.368448 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:09:57.368579 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:09:57.380561 (+ 11982us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:09:57.380567 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:09:57.380570 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:09:57.380606 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:09:57.383458 (+ 2852us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:09:57.384431 (+ 973us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:09:57.384437 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:09:57.384440 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:09:57.384613 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:09:57.385080 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:09:57.385084 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:09:57.385086 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:09:57.385537 (+ 451us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:09:57.385680 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0923 20:09:57.386647 (+ 967us) server_negotiation.cc:299] Negotiation successful
0923 20:09:57.386827 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260923 20:09:57.388756 10341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:09:57.361004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.190:36595 (local address 127.1.170.130:58025)
0923 20:09:57.361680 (+ 676us) negotiation.cc:107] Waiting for socket to connect
0923 20:09:57.361720 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0923 20:09:57.363722 (+ 2002us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:09:57.364403 (+ 681us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:09:57.364410 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:09:57.365007 (+ 597us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:09:57.365544 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:09:57.365560 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:09:57.366638 (+ 1078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:09:57.366642 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:09:57.367864 (+ 1222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:09:57.367876 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:09:57.368666 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:09:57.368673 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:09:57.368908 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:09:57.377248 (+ 8340us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:09:57.377292 (+ 44us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:09:57.380261 (+ 2969us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:09:57.383639 (+ 3378us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:09:57.383646 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:09:57.383661 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:09:57.384175 (+ 514us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:09:57.384739 (+ 564us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:09:57.384743 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:09:57.384745 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:09:57.384935 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:09:57.385713 (+ 778us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:09:57.385724 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:09:57.386244 (+ 520us) client_negotiation.cc:769] Sending connection context
0923 20:09:57.386572 (+ 328us) client_negotiation.cc:240] Negotiation successful
0923 20:09:57.387235 (+ 663us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":473,"thread_start_us":278,"threads_started":1}
I20260923 20:09:57.390339 10339 heartbeater.cc:347] Connected to a master server at 127.1.170.190:36595
I20260923 20:09:57.390645 10339 heartbeater.cc:464] Registering TS with master...
I20260923 20:09:57.391355 10339 heartbeater.cc:511] Master 127.1.170.190:36595 requested a full tablet report, sending...
I20260923 20:09:57.393160 10016 ts_manager.cc:194] Registered new tserver with Master: 0790313c2d224e8fb772982e48a057ab (127.1.170.130:45381)
I20260923 20:09:57.394068 10016 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.170.130@KRBTEST.COM'} at 127.1.170.130:58025
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.170.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194121577027-1706-0/krb5kdc/kudu_127.1.170.131.keytab.
Entry for principal kudu/127.1.170.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194121577027-1706-0/krb5kdc/kudu_127.1.170.131.keytab.
I20260923 20:09:57.415519 1706 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqUuku6/build/asan/bin/kudu
/tmp/dist-test-taskqUuku6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194121577027-1706-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194121577027-1706-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194121577027-1706-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194121577027-1706-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.170.131:0
--local_ip_for_outbound_sockets=127.1.170.131
--webserver_interface=127.1.170.131
--webserver_port=0
--tserver_master_addrs=127.1.170.190:36595
--keytab_file=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194121577027-1706-0/krb5kdc/kudu_127.1.170.131.keytab
--principal=kudu/127.1.170.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.170.148:38359
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194121577027-1706-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194121577027-1706-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194121577027-1706-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:09:57.735528 10344 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:09:57.735805 10344 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:09:57.735975 10344 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:09:57.743359 10344 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:09:57.743480 10344 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:09:57.743654 10344 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.170.131
I20260923 20:09:57.753495 10344 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.170.148:38359
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194121577027-1706-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194121577027-1706-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194121577027-1706-0/krb5kdc/kudu_127.1.170.131.keytab
--principal=kudu/127.1.170.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.170.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194121577027-1706-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.170.131
--webserver_port=0
--tserver_master_addrs=127.1.170.190:36595
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.170.131
--log_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194121577027-1706-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:09:57.754801 10344 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:09:57.756158 10344 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:09:57.771881 10351 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:09:57.771950 10350 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:09:57.772538 10353 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:09:57.775548 10344 server_base.cc:1034] running on GCE node
I20260923 20:09:57.776176 10344 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:09:57.777091 10344 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:09:57.778291 10344 hybrid_clock.cc:648] HybridClock initialized: now 1790194197778223 us; error 53 us; skew 500 ppm
I20260923 20:09:57.778614 10344 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:09:57 dist-test-slave-th4g krb5kdc[9971](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194197, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.170.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:09:57.781697 10344 init.cc:377] Logged in from keytab as kudu/127.1.170.131@KRBTEST.COM (short username kudu)
I20260923 20:09:57.783381 10344 webserver.cc:460] Webserver started at http://127.1.170.131:38791/ using document root <none> and password file <none>
I20260923 20:09:57.784103 10344 fs_manager.cc:359] Metadata directory not provided
I20260923 20:09:57.784221 10344 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:09:57.784524 10344 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:09:57.787384 10344 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194121577027-1706-0/minicluster-data/ts-2/data/instance:
uuid: "61453ab85a0049199388b617ec073676"
format_stamp: "Formatted at 2026-09-23 20:09:57 on dist-test-slave-th4g"
server_key: "261bb016a3c0e268275f8b069897560b"
server_key_iv: "77aa37704abeecff83d88c2a7adc6c12"
server_key_version: "encryptionkey@0"
I20260923 20:09:57.788164 10344 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194121577027-1706-0/minicluster-data/ts-2/wal/instance:
uuid: "61453ab85a0049199388b617ec073676"
format_stamp: "Formatted at 2026-09-23 20:09:57 on dist-test-slave-th4g"
server_key: "261bb016a3c0e268275f8b069897560b"
server_key_iv: "77aa37704abeecff83d88c2a7adc6c12"
server_key_version: "encryptionkey@0"
I20260923 20:09:57.793227 10344 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.005s
I20260923 20:09:57.797314 10360 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:09:57.798962 10344 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 20:09:57.799170 10344 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194121577027-1706-0/minicluster-data/ts-2/data,/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194121577027-1706-0/minicluster-data/ts-2/wal
uuid: "61453ab85a0049199388b617ec073676"
format_stamp: "Formatted at 2026-09-23 20:09:57 on dist-test-slave-th4g"
server_key: "261bb016a3c0e268275f8b069897560b"
server_key_iv: "77aa37704abeecff83d88c2a7adc6c12"
server_key_version: "encryptionkey@0"
I20260923 20:09:57.799371 10344 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194121577027-1706-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194121577027-1706-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194121577027-1706-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:09:57.845865 10344 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:09:57.850675 10344 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:09:57.850992 10344 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:09:57.852025 10344 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 20:09:57.853435 10344 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 20:09:57.853541 10344 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 20:09:57.853698 10344 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 20:09:57.853770 10344 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 20:09:57.881914 10205 heartbeater.cc:503] Master 127.1.170.190:36595 was elected leader, sending a full tablet report...
I20260923 20:09:57.890952 10344 rpc_server.cc:307] RPC server started. Bound to: 127.1.170.131:46717
I20260923 20:09:57.891093 10472 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.170.131:46717 every 8 connection(s)
I20260923 20:09:57.892437 10344 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194121577027-1706-0/minicluster-data/ts-2/data/info.pb
I20260923 20:09:57.898551 1706 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqUuku6/build/asan/bin/kudu as pid 10344
I20260923 20:09:57.898778 1706 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194121577027-1706-0/minicluster-data/ts-2/wal/instance
I20260923 20:09:57.899291 1706 external_mini_cluster.cc:1428] Setting key 0c319a3c89eac8420d75a12cb2bd7c21
Sep 23 20:09:57 dist-test-slave-th4g krb5kdc[9971](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194197, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.170.131@KRBTEST.COM for kudu/127.1.170.190@KRBTEST.COM
I20260923 20:09:57.913865 10475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:09:57.895093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.170.131:56537 (local address 127.1.170.190:36595)
0923 20:09:57.895601 (+ 508us) server_negotiation.cc:206] Beginning negotiation
0923 20:09:57.895618 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0923 20:09:57.896834 (+ 1216us) server_negotiation.cc:407] Connection header received
0923 20:09:57.898945 (+ 2111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:09:57.898952 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:09:57.899037 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:09:57.899162 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:09:57.900951 (+ 1789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:09:57.902093 (+ 1142us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:09:57.903926 (+ 1833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:09:57.904319 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:09:57.904390 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:09:57.908116 (+ 3726us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:09:57.908121 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:09:57.908125 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:09:57.908162 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:09:57.910663 (+ 2501us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:09:57.911404 (+ 741us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:09:57.911410 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:09:57.911416 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:09:57.911538 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:09:57.911963 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:09:57.911970 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:09:57.911973 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:09:57.912241 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:09:57.912406 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0923 20:09:57.913241 (+ 835us) server_negotiation.cc:299] Negotiation successful
0923 20:09:57.913464 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":324,"thread_start_us":214,"threads_started":1}
I20260923 20:09:57.914894 10476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:09:57.895681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.190:36595 (local address 127.1.170.131:56537)
0923 20:09:57.896629 (+ 948us) negotiation.cc:107] Waiting for socket to connect
0923 20:09:57.896681 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0923 20:09:57.898586 (+ 1905us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:09:57.899499 (+ 913us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:09:57.899507 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:09:57.900109 (+ 602us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:09:57.900781 (+ 672us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:09:57.900799 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:09:57.902249 (+ 1450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:09:57.902255 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:09:57.903705 (+ 1450us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:09:57.903717 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:09:57.904639 (+ 922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:09:57.904643 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:09:57.904845 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:09:57.905505 (+ 660us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:09:57.905526 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:09:57.907892 (+ 2366us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:09:57.910811 (+ 2919us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:09:57.910821 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:09:57.910847 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:09:57.911274 (+ 427us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:09:57.911659 (+ 385us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:09:57.911664 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:09:57.911668 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:09:57.911851 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:09:57.912371 (+ 520us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:09:57.912381 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:09:57.912806 (+ 425us) client_negotiation.cc:769] Sending connection context
0923 20:09:57.913167 (+ 361us) client_negotiation.cc:240] Negotiation successful
0923 20:09:57.913698 (+ 531us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":656,"thread_start_us":343,"threads_started":1}
I20260923 20:09:57.916487 10473 heartbeater.cc:347] Connected to a master server at 127.1.170.190:36595
I20260923 20:09:57.916818 10473 heartbeater.cc:464] Registering TS with master...
I20260923 20:09:57.918286 10473 heartbeater.cc:511] Master 127.1.170.190:36595 requested a full tablet report, sending...
I20260923 20:09:57.920086 10016 ts_manager.cc:194] Registered new tserver with Master: 61453ab85a0049199388b617ec073676 (127.1.170.131:46717)
I20260923 20:09:57.920914 10016 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.170.131@KRBTEST.COM'} at 127.1.170.131:56537
I20260923 20:09:57.928491 1706 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.1.170.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.1.170.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.1.170.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194121577027-1706-0/krb5kdc/oryx_127.1.170.129.keytab.
Entry for principal oryx/127.1.170.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194121577027-1706-0/krb5kdc/oryx_127.1.170.129.keytab.
WARNING: no policy specified for oryx/127.1.170.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.1.170.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.1.170.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194121577027-1706-0/krb5kdc/oryx_127.1.170.130.keytab.
Entry for principal oryx/127.1.170.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194121577027-1706-0/krb5kdc/oryx_127.1.170.130.keytab.
WARNING: no policy specified for oryx/127.1.170.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.1.170.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.1.170.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194121577027-1706-0/krb5kdc/oryx_127.1.170.131.keytab.
Entry for principal oryx/127.1.170.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194121577027-1706-0/krb5kdc/oryx_127.1.170.131.keytab.
I20260923 20:09:58.082201 1706 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskqUuku6/build/asan/bin/kudu pid 10077
Sep 23 20:09:58 dist-test-slave-th4g krb5kdc[9971](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194195, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.170.129@KRBTEST.COM
I20260923 20:09:58.102519 10484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:09:58.083446 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41738 (local address 127.1.170.129:44063)
0923 20:09:58.084114 (+ 668us) server_negotiation.cc:206] Beginning negotiation
0923 20:09:58.084121 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:09:58.084148 (+ 27us) server_negotiation.cc:407] Connection header received
0923 20:09:58.084208 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:09:58.084215 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:09:58.084426 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:09:58.084561 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:09:58.086274 (+ 1713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:09:58.087802 (+ 1528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:09:58.089402 (+ 1600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:09:58.090146 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:09:58.090283 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:09:58.095797 (+ 5514us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:09:58.095813 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:09:58.095830 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:09:58.095879 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:09:58.099393 (+ 3514us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:09:58.100193 (+ 800us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:09:58.100200 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:09:58.100203 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:09:58.100374 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:09:58.100738 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:09:58.100742 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:09:58.100744 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:09:58.101131 (+ 387us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:09:58.101500 (+ 369us) server_negotiation.cc:1035] Waiting for connection context
0923 20:09:58.101798 (+ 298us) server_negotiation.cc:299] Negotiation successful
0923 20:09:58.102115 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":493,"thread_start_us":364,"threads_started":1}
I20260923 20:09:58.103859 10200 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:09:58.189225 1706 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqUuku6/build/asan/bin/kudu with pid 10077
I20260923 20:09:58.206698 1706 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskqUuku6/build/asan/bin/kudu pid 10211
Sep 23 20:09:58 dist-test-slave-th4g krb5kdc[9971](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194195, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.170.130@KRBTEST.COM
I20260923 20:09:58.226084 10486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:09:58.207654 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40792 (local address 127.1.170.130:45381)
0923 20:09:58.208422 (+ 768us) server_negotiation.cc:206] Beginning negotiation
0923 20:09:58.208433 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0923 20:09:58.208482 (+ 49us) server_negotiation.cc:407] Connection header received
0923 20:09:58.208569 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:09:58.208577 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:09:58.208868 (+ 291us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:09:58.209175 (+ 307us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:09:58.210791 (+ 1616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:09:58.211875 (+ 1084us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:09:58.213132 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:09:58.213742 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:09:58.213960 (+ 218us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:09:58.219421 (+ 5461us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:09:58.219436 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:09:58.219454 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:09:58.219495 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:09:58.222621 (+ 3126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:09:58.223385 (+ 764us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:09:58.223393 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:09:58.223395 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:09:58.223526 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:09:58.224145 (+ 619us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:09:58.224148 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:09:58.224150 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:09:58.224503 (+ 353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:09:58.224777 (+ 274us) server_negotiation.cc:1035] Waiting for connection context
0923 20:09:58.225183 (+ 406us) server_negotiation.cc:299] Negotiation successful
0923 20:09:58.225491 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":573,"thread_start_us":377,"threads_started":1}
I20260923 20:09:58.227419 10334 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:09:58.311492 1706 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqUuku6/build/asan/bin/kudu with pid 10211
I20260923 20:09:58.327613 1706 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskqUuku6/build/asan/bin/kudu pid 10344
Sep 23 20:09:58 dist-test-slave-th4g krb5kdc[9971](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194195, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.170.131@KRBTEST.COM
I20260923 20:09:58.344772 10488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:09:58.328576 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33876 (local address 127.1.170.131:46717)
0923 20:09:58.329094 (+ 518us) server_negotiation.cc:206] Beginning negotiation
0923 20:09:58.329101 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:09:58.329131 (+ 30us) server_negotiation.cc:407] Connection header received
0923 20:09:58.329201 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:09:58.329208 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:09:58.329475 (+ 267us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:09:58.329604 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:09:58.331104 (+ 1500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:09:58.332137 (+ 1033us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:09:58.333472 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:09:58.334119 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:09:58.334229 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:09:58.338864 (+ 4635us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:09:58.338879 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:09:58.338899 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:09:58.338938 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:09:58.342001 (+ 3063us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:09:58.342719 (+ 718us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:09:58.342726 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:09:58.342728 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:09:58.342874 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:09:58.343285 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:09:58.343288 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:09:58.343290 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:09:58.343633 (+ 343us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:09:58.343895 (+ 262us) server_negotiation.cc:1035] Waiting for connection context
0923 20:09:58.344156 (+ 261us) server_negotiation.cc:299] Negotiation successful
0923 20:09:58.344352 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":376,"thread_start_us":238,"threads_started":1}
I20260923 20:09:58.346205 10468 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:09:58.432582 1706 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqUuku6/build/asan/bin/kudu with pid 10344
I20260923 20:09:58.448624 1706 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskqUuku6/build/asan/bin/kudu pid 9984
I20260923 20:09:58.449939 10046 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:09:58.554530 1706 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqUuku6/build/asan/bin/kudu with pid 9984
I20260923 20:09:58.576502 1706 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqUuku6/build/asan/bin/kudu
/tmp/dist-test-taskqUuku6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194121577027-1706-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194121577027-1706-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194121577027-1706-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194121577027-1706-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.170.190:36595
--webserver_interface=127.1.170.190
--webserver_port=41157
--keytab_file=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194121577027-1706-0/krb5kdc/kudu_127.1.170.190.keytab
--principal=kudu/127.1.170.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.170.148:38359
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.170.190:36595
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194121577027-1706-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194121577027-1706-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194121577027-1706-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:09:58.919018 10491 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:09:58.919420 10491 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:09:58.919588 10491 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:09:58.927520 10491 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 20:09:58.927698 10491 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:09:58.927820 10491 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:09:58.927922 10491 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 20:09:58.928012 10491 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 20:09:58.938931 10491 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.170.148:38359
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194121577027-1706-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194121577027-1706-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.170.190:36595
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194121577027-1706-0/krb5kdc/kudu_127.1.170.190.keytab
--principal=kudu/127.1.170.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.170.190:36595
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194121577027-1706-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.170.190
--webserver_port=41157
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194121577027-1706-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:09:58.940369 10491 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:09:58.941680 10491 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:09:58.955022 10496 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:09:58.955041 10497 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:09:58.954995 10499 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:09:58.956022 10491 server_base.cc:1034] running on GCE node
I20260923 20:09:58.956863 10491 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:09:58.958531 10491 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:09:58.959841 10491 hybrid_clock.cc:648] HybridClock initialized: now 1790194198959739 us; error 79 us; skew 500 ppm
I20260923 20:09:58.960253 10491 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:09:58 dist-test-slave-th4g krb5kdc[9971](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194198, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.170.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:09:58.963529 10491 init.cc:377] Logged in from keytab as kudu/127.1.170.190@KRBTEST.COM (short username kudu)
I20260923 20:09:58.965476 10491 webserver.cc:460] Webserver started at http://127.1.170.190:41157/ using document root <none> and password file <none>
I20260923 20:09:58.966255 10491 fs_manager.cc:359] Metadata directory not provided
I20260923 20:09:58.966372 10491 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:09:58.972695 10491 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260923 20:09:58.976970 10506 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:09:58.979156 10491 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260923 20:09:58.979399 10491 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194121577027-1706-0/minicluster-data/master-0/data,/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194121577027-1706-0/minicluster-data/master-0/wal
uuid: "40fdd4262e6c4efa855ec33d169d0953"
format_stamp: "Formatted at 2026-09-23 20:09:56 on dist-test-slave-th4g"
server_key: "71ce9fa5a34c53992a56b3f6ac69b42f"
server_key_iv: "22b4be7949884a5dbc210bfba42acc40"
server_key_version: "encryptionkey@0"
I20260923 20:09:58.980481 10491 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194121577027-1706-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194121577027-1706-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194121577027-1706-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:09:59.006623 10491 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:09:59.011543 10491 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:09:59.011827 10491 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:09:59.033627 10491 rpc_server.cc:307] RPC server started. Bound to: 127.1.170.190:36595
I20260923 20:09:59.033674 10557 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.170.190:36595 every 8 connection(s)
I20260923 20:09:59.035266 10491 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194121577027-1706-0/minicluster-data/master-0/data/info.pb
I20260923 20:09:59.040850 1706 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqUuku6/build/asan/bin/kudu as pid 10491
I20260923 20:09:59.041793 1706 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqUuku6/build/asan/bin/kudu
/tmp/dist-test-taskqUuku6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194121577027-1706-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194121577027-1706-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194121577027-1706-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194121577027-1706-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.170.129:44063
--local_ip_for_outbound_sockets=127.1.170.129
--tserver_master_addrs=127.1.170.190:36595
--webserver_port=41889
--webserver_interface=127.1.170.129
--keytab_file=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194121577027-1706-0/krb5kdc/kudu_127.1.170.129.keytab
--principal=kudu/127.1.170.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.170.148:38359
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.1.170.129
--keytab_file=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194121577027-1706-0/krb5kdc/oryx_127.1.170.129.keytab with env {KRB5CCNAME=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194121577027-1706-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194121577027-1706-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194121577027-1706-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260923 20:09:59.042932 10558 sys_catalog.cc:263] Verifying existing consensus state
I20260923 20:09:59.048628 10558 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 40fdd4262e6c4efa855ec33d169d0953: Bootstrap starting.
I20260923 20:09:59.073253 10558 log.cc:824] T 00000000000000000000000000000000 P 40fdd4262e6c4efa855ec33d169d0953: Log is configured to *not* fsync() on all Append() calls
I20260923 20:09:59.083986 10558 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 40fdd4262e6c4efa855ec33d169d0953: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260923 20:09:59.084846 10558 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 40fdd4262e6c4efa855ec33d169d0953: Bootstrap complete.
I20260923 20:09:59.095603 10558 raft_consensus.cc:348] T 00000000000000000000000000000000 P 40fdd4262e6c4efa855ec33d169d0953 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40fdd4262e6c4efa855ec33d169d0953" member_type: VOTER last_known_addr { host: "127.1.170.190" port: 36595 } }
I20260923 20:09:59.096580 10558 raft_consensus.cc:729] T 00000000000000000000000000000000 P 40fdd4262e6c4efa855ec33d169d0953 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 40fdd4262e6c4efa855ec33d169d0953, State: Initialized, Role: FOLLOWER
I20260923 20:09:59.097317 10558 consensus_queue.cc:260] T 00000000000000000000000000000000 P 40fdd4262e6c4efa855ec33d169d0953 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40fdd4262e6c4efa855ec33d169d0953" member_type: VOTER last_known_addr { host: "127.1.170.190" port: 36595 } }
I20260923 20:09:59.097569 10558 raft_consensus.cc:388] T 00000000000000000000000000000000 P 40fdd4262e6c4efa855ec33d169d0953 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 20:09:59.097687 10558 raft_consensus.cc:482] T 00000000000000000000000000000000 P 40fdd4262e6c4efa855ec33d169d0953 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 20:09:59.097915 10558 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 40fdd4262e6c4efa855ec33d169d0953 [term 1 FOLLOWER]: Advancing to term 2
I20260923 20:09:59.102072 10558 raft_consensus.cc:504] T 00000000000000000000000000000000 P 40fdd4262e6c4efa855ec33d169d0953 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40fdd4262e6c4efa855ec33d169d0953" member_type: VOTER last_known_addr { host: "127.1.170.190" port: 36595 } }
I20260923 20:09:59.102643 10558 leader_election.cc:302] T 00000000000000000000000000000000 P 40fdd4262e6c4efa855ec33d169d0953 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 40fdd4262e6c4efa855ec33d169d0953; no voters:
I20260923 20:09:59.103322 10558 leader_election.cc:288] T 00000000000000000000000000000000 P 40fdd4262e6c4efa855ec33d169d0953 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260923 20:09:59.103461 10562 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 40fdd4262e6c4efa855ec33d169d0953 [term 2 FOLLOWER]: Leader election won for term 2
I20260923 20:09:59.104175 10562 raft_consensus.cc:686] T 00000000000000000000000000000000 P 40fdd4262e6c4efa855ec33d169d0953 [term 2 LEADER]: Becoming Leader. State: Replica: 40fdd4262e6c4efa855ec33d169d0953, State: Running, Role: LEADER
I20260923 20:09:59.104971 10562 consensus_queue.cc:237] T 00000000000000000000000000000000 P 40fdd4262e6c4efa855ec33d169d0953 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40fdd4262e6c4efa855ec33d169d0953" member_type: VOTER last_known_addr { host: "127.1.170.190" port: 36595 } }
I20260923 20:09:59.106536 10558 sys_catalog.cc:564] T 00000000000000000000000000000000 P 40fdd4262e6c4efa855ec33d169d0953 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 20:09:59.107539 10564 sys_catalog.cc:455] T 00000000000000000000000000000000 P 40fdd4262e6c4efa855ec33d169d0953 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "40fdd4262e6c4efa855ec33d169d0953" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40fdd4262e6c4efa855ec33d169d0953" member_type: VOTER last_known_addr { host: "127.1.170.190" port: 36595 } } }
I20260923 20:09:59.108315 10564 sys_catalog.cc:458] T 00000000000000000000000000000000 P 40fdd4262e6c4efa855ec33d169d0953 [sys.catalog]: This master's current role is: LEADER
I20260923 20:09:59.107956 10563 sys_catalog.cc:455] T 00000000000000000000000000000000 P 40fdd4262e6c4efa855ec33d169d0953 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 40fdd4262e6c4efa855ec33d169d0953. Latest consensus state: current_term: 2 leader_uuid: "40fdd4262e6c4efa855ec33d169d0953" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40fdd4262e6c4efa855ec33d169d0953" member_type: VOTER last_known_addr { host: "127.1.170.190" port: 36595 } } }
I20260923 20:09:59.108851 10563 sys_catalog.cc:458] T 00000000000000000000000000000000 P 40fdd4262e6c4efa855ec33d169d0953 [sys.catalog]: This master's current role is: LEADER
I20260923 20:09:59.111207 10567 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 20:09:59.121407 10567 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 20:09:59.123489 10567 catalog_manager.cc:1259] Loaded cluster ID: 765d13068e3e4084a16f3875bac23574
I20260923 20:09:59.123695 10567 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 20:09:59.128646 10567 catalog_manager.cc:1504] Loading token signing keys...
I20260923 20:09:59.130406 10567 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 40fdd4262e6c4efa855ec33d169d0953: Loaded TSK: 0
I20260923 20:09:59.132159 10567 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260923 20:09:59.452169 10560 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:09:59.452749 10560 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:09:59.453145 10560 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:09:59.462395 10560 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:09:59.462568 10560 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:09:59.462760 10560 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.170.129
I20260923 20:09:59.475875 10560 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.170.148:38359
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194121577027-1706-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194121577027-1706-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194121577027-1706-0/krb5kdc/oryx_127.1.170.129.keytab
--principal=oryx/127.1.170.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.170.129:44063
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194121577027-1706-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.170.129
--webserver_port=41889
--tserver_master_addrs=127.1.170.190:36595
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.170.129
--log_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194121577027-1706-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:09:59.477381 10560 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:09:59.479585 10560 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:09:59.496495 10585 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:09:59.496510 10584 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:09:59.497812 10587 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:09:59.504328 10560 server_base.cc:1034] running on GCE node
I20260923 20:09:59.505077 10560 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:09:59.506822 10560 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:09:59.508090 10560 hybrid_clock.cc:648] HybridClock initialized: now 1790194199508026 us; error 77 us; skew 500 ppm
I20260923 20:09:59.508564 10560 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:09:59 dist-test-slave-th4g krb5kdc[9971](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194199, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.170.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:09:59.513708 10560 init.cc:377] Logged in from keytab as oryx/127.1.170.129@KRBTEST.COM (short username oryx)
I20260923 20:09:59.515605 10560 webserver.cc:460] Webserver started at http://127.1.170.129:41889/ using document root <none> and password file <none>
I20260923 20:09:59.516446 10560 fs_manager.cc:359] Metadata directory not provided
I20260923 20:09:59.516642 10560 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:09:59.523216 10560 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260923 20:09:59.526518 10594 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:09:59.528820 10560 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.002s
I20260923 20:09:59.529296 10560 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194121577027-1706-0/minicluster-data/ts-0/data,/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194121577027-1706-0/minicluster-data/ts-0/wal
uuid: "0bfbaf6365cf4baea93de6401ebcc278"
format_stamp: "Formatted at 2026-09-23 20:09:56 on dist-test-slave-th4g"
server_key: "3c1fdc0fae70f1ca246d8297da00edbc"
server_key_iv: "cad483cd70c9dba3d3a06c11c915ccf1"
server_key_version: "encryptionkey@0"
I20260923 20:09:59.530963 10560 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194121577027-1706-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194121577027-1706-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194121577027-1706-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:09:59.550827 10560 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:09:59.555536 10560 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:09:59.555837 10560 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:09:59.556880 10560 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 20:09:59.558374 10560 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 20:09:59.558496 10560 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 20:09:59.558632 10560 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 20:09:59.558723 10560 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 20:09:59.604084 10560 rpc_server.cc:307] RPC server started. Bound to: 127.1.170.129:44063
I20260923 20:09:59.604130 10707 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.170.129:44063 every 8 connection(s)
I20260923 20:09:59.605551 10560 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194121577027-1706-0/minicluster-data/ts-0/data/info.pb
I20260923 20:09:59.612810 1706 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqUuku6/build/asan/bin/kudu as pid 10560
I20260923 20:09:59.613580 1706 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqUuku6/build/asan/bin/kudu
/tmp/dist-test-taskqUuku6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194121577027-1706-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194121577027-1706-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194121577027-1706-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194121577027-1706-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.170.130:45381
--local_ip_for_outbound_sockets=127.1.170.130
--tserver_master_addrs=127.1.170.190:36595
--webserver_port=37077
--webserver_interface=127.1.170.130
--keytab_file=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194121577027-1706-0/krb5kdc/kudu_127.1.170.130.keytab
--principal=kudu/127.1.170.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.170.148:38359
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.1.170.130
--keytab_file=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194121577027-1706-0/krb5kdc/oryx_127.1.170.130.keytab with env {KRB5CCNAME=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194121577027-1706-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194121577027-1706-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194121577027-1706-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 23 20:09:59 dist-test-slave-th4g krb5kdc[9971](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.170.129@KRBTEST.COM for oryx/127.1.170.190@KRBTEST.COM, Server not found in Kerberos database
Sep 23 20:09:59 dist-test-slave-th4g krb5kdc[9971](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.170.129@KRBTEST.COM for oryx/127.1.170.190@KRBTEST.COM, Server not found in Kerberos database
W20260923 20:09:59.973039 10712 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:09:59.973357 10712 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:09:59.973505 10712 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:09:59.982439 10712 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:09:59.982581 10712 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:09:59.982772 10712 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.170.130
I20260923 20:09:59.994081 10712 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.170.148:38359
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194121577027-1706-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194121577027-1706-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194121577027-1706-0/krb5kdc/oryx_127.1.170.130.keytab
--principal=oryx/127.1.170.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.170.130:45381
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194121577027-1706-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.170.130
--webserver_port=37077
--tserver_master_addrs=127.1.170.190:36595
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.170.130
--log_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194121577027-1706-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:09:59.996680 10712 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:09:59.998899 10712 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:10:00.021911 10718 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:10:00.022311 10719 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:10:00.023633 10721 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:10:00.024031 10712 server_base.cc:1034] running on GCE node
I20260923 20:10:00.025228 10712 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:10:00.027302 10712 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:10:00.028781 10712 hybrid_clock.cc:648] HybridClock initialized: now 1790194200028633 us; error 65 us; skew 500 ppm
I20260923 20:10:00.029466 10712 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:10:00 dist-test-slave-th4g krb5kdc[9971](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194200, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.170.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:10:00.035890 10712 init.cc:377] Logged in from keytab as oryx/127.1.170.130@KRBTEST.COM (short username oryx)
I20260923 20:10:00.039103 10712 webserver.cc:460] Webserver started at http://127.1.170.130:37077/ using document root <none> and password file <none>
I20260923 20:10:00.040445 10712 fs_manager.cc:359] Metadata directory not provided
I20260923 20:10:00.040666 10712 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:10:00.052663 10712 fs_manager.cc:709] Time spent opening directory manager: real 0.008s user 0.006s sys 0.003s
I20260923 20:10:00.058254 10728 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:10:00.060807 10712 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.003s sys 0.001s
I20260923 20:10:00.061127 10712 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194121577027-1706-0/minicluster-data/ts-1/data,/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194121577027-1706-0/minicluster-data/ts-1/wal
uuid: "0790313c2d224e8fb772982e48a057ab"
format_stamp: "Formatted at 2026-09-23 20:09:57 on dist-test-slave-th4g"
server_key: "540df7a8190cc9cc4d3533f2d0747bf9"
server_key_iv: "e00c51e18cd99695b021a3166361d0e8"
server_key_version: "encryptionkey@0"
I20260923 20:10:00.063350 10712 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194121577027-1706-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194121577027-1706-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194121577027-1706-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:10:00.113902 10712 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:10:00.118587 10712 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:10:00.118880 10712 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:10:00.119944 10712 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 20:10:00.121419 10712 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 20:10:00.121551 10712 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 20:10:00.121688 10712 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 20:10:00.121784 10712 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 20:10:00.164531 10712 rpc_server.cc:307] RPC server started. Bound to: 127.1.170.130:45381
I20260923 20:10:00.164573 10840 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.170.130:45381 every 8 connection(s)
I20260923 20:10:00.166002 10712 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194121577027-1706-0/minicluster-data/ts-1/data/info.pb
I20260923 20:10:00.172626 1706 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqUuku6/build/asan/bin/kudu as pid 10712
I20260923 20:10:00.173671 1706 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqUuku6/build/asan/bin/kudu
/tmp/dist-test-taskqUuku6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194121577027-1706-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194121577027-1706-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194121577027-1706-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194121577027-1706-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.170.131:46717
--local_ip_for_outbound_sockets=127.1.170.131
--tserver_master_addrs=127.1.170.190:36595
--webserver_port=38791
--webserver_interface=127.1.170.131
--keytab_file=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194121577027-1706-0/krb5kdc/kudu_127.1.170.131.keytab
--principal=kudu/127.1.170.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.170.148:38359
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.1.170.131
--keytab_file=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194121577027-1706-0/krb5kdc/oryx_127.1.170.131.keytab with env {KRB5CCNAME=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194121577027-1706-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194121577027-1706-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194121577027-1706-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 23 20:10:00 dist-test-slave-th4g krb5kdc[9971](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.170.130@KRBTEST.COM for oryx/127.1.170.190@KRBTEST.COM, Server not found in Kerberos database
Sep 23 20:10:00 dist-test-slave-th4g krb5kdc[9971](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.170.130@KRBTEST.COM for oryx/127.1.170.190@KRBTEST.COM, Server not found in Kerberos database
W20260923 20:10:00.638731 10845 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:10:00.639354 10845 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:10:00.639593 10845 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:10:00.658986 10845 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:10:00.659217 10845 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:10:00.659533 10845 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.170.131
I20260923 20:10:00.685011 10845 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.170.148:38359
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194121577027-1706-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194121577027-1706-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194121577027-1706-0/krb5kdc/oryx_127.1.170.131.keytab
--principal=oryx/127.1.170.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.170.131:46717
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194121577027-1706-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.170.131
--webserver_port=38791
--tserver_master_addrs=127.1.170.190:36595
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.170.131
--log_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194121577027-1706-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:10:00.687584 10845 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:10:00.689982 10845 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:10:00.715066 10851 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:10:00.718750 10852 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:10:00.719354 10855 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:10:00.720599 10845 server_base.cc:1034] running on GCE node
I20260923 20:10:00.721698 10845 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:10:00.723394 10845 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:10:00.724912 10845 hybrid_clock.cc:648] HybridClock initialized: now 1790194200724803 us; error 79 us; skew 500 ppm
I20260923 20:10:00.725495 10845 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:10:00 dist-test-slave-th4g krb5kdc[9971](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194200, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.170.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:10:00.731576 10845 init.cc:377] Logged in from keytab as oryx/127.1.170.131@KRBTEST.COM (short username oryx)
I20260923 20:10:00.735018 10845 webserver.cc:460] Webserver started at http://127.1.170.131:38791/ using document root <none> and password file <none>
I20260923 20:10:00.736323 10845 fs_manager.cc:359] Metadata directory not provided
I20260923 20:10:00.736541 10845 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:10:00.746927 10845 fs_manager.cc:709] Time spent opening directory manager: real 0.007s user 0.007s sys 0.000s
I20260923 20:10:00.751825 10861 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:10:00.754398 10845 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.006s sys 0.000s
I20260923 20:10:00.754680 10845 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194121577027-1706-0/minicluster-data/ts-2/data,/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194121577027-1706-0/minicluster-data/ts-2/wal
uuid: "61453ab85a0049199388b617ec073676"
format_stamp: "Formatted at 2026-09-23 20:09:57 on dist-test-slave-th4g"
server_key: "261bb016a3c0e268275f8b069897560b"
server_key_iv: "77aa37704abeecff83d88c2a7adc6c12"
server_key_version: "encryptionkey@0"
I20260923 20:10:00.756639 10845 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194121577027-1706-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194121577027-1706-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194121577027-1706-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:10:00.798231 10845 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:10:00.803714 10845 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:10:00.803994 10845 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:10:00.805218 10845 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 20:10:00.806958 10845 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 20:10:00.807066 10845 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 20:10:00.807163 10845 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 20:10:00.807232 10845 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 20:10:00.861265 10845 rpc_server.cc:307] RPC server started. Bound to: 127.1.170.131:46717
I20260923 20:10:00.861567 10974 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.170.131:46717 every 8 connection(s)
I20260923 20:10:00.863263 10845 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194121577027-1706-0/minicluster-data/ts-2/data/info.pb
I20260923 20:10:00.865453 1706 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqUuku6/build/asan/bin/kudu as pid 10845
Sep 23 20:10:00 dist-test-slave-th4g krb5kdc[9971](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.170.131@KRBTEST.COM for oryx/127.1.170.190@KRBTEST.COM, Server not found in Kerberos database
Sep 23 20:10:00 dist-test-slave-th4g krb5kdc[9971](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.170.131@KRBTEST.COM for oryx/127.1.170.190@KRBTEST.COM, Server not found in Kerberos database
I20260923 20:10:00.915205 10978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:10:00.867013 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39472 (local address 127.1.170.190:36595)
0923 20:10:00.882910 (+ 15897us) server_negotiation.cc:206] Beginning negotiation
0923 20:10:00.882921 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0923 20:10:00.895247 (+ 12326us) server_negotiation.cc:407] Connection header received
0923 20:10:00.895428 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:10:00.895433 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:10:00.895508 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:10:00.895622 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:10:00.897579 (+ 1957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:10:00.898808 (+ 1229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:10:00.900606 (+ 1798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:10:00.901224 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:10:00.901321 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:10:00.906066 (+ 4745us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:10:00.906077 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:10:00.906100 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:10:00.906154 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:10:00.910690 (+ 4536us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:10:00.911357 (+ 667us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:10:00.911368 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:10:00.911381 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:10:00.911581 (+ 200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:10:00.911963 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:10:00.911971 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:10:00.911978 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:10:00.912687 (+ 709us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:10:00.912984 (+ 297us) server_negotiation.cc:1035] Waiting for connection context
0923 20:10:00.913487 (+ 503us) server_negotiation.cc:299] Negotiation successful
0923 20:10:00.914013 (+ 526us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15690,"thread_start_us":15909,"threads_started":1}
W20260923 20:10:00.993736 10710 sasl_common.cc:323] Time spent calling sasl_client_start(): real 1.369s user 0.002s sys 0.013s
W20260923 20:10:00.996196 10711 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 20:09:59.609383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.170.129:34355 (local address 127.1.170.190:36595)
0923 20:09:59.610360 (+ 977us) server_negotiation.cc:206] Beginning negotiation
0923 20:09:59.610372 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0923 20:09:59.610525 (+ 153us) server_negotiation.cc:407] Connection header received
0923 20:09:59.612766 (+ 2241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:09:59.612779 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:09:59.613221 (+ 442us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:09:59.613604 (+ 383us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:09:59.617647 (+ 4043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:09:59.619530 (+ 1883us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:09:59.621264 (+ 1734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:09:59.622437 (+ 1173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:09:59.622773 (+ 336us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:10:00.995640 (+1372867us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.170.129:34355: BlockingRecv error: recv got EOF from 127.1.170.129:34355 (error 108)
Metrics: {"server-negotiator.queue_time_us":704,"thread_start_us":467,"threads_started":1}
I20260923 20:10:00.998646 10710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:09:59.608859 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.190:36595 (local address 127.1.170.129:34355)
0923 20:09:59.610193 (+ 1334us) negotiation.cc:107] Waiting for socket to connect
0923 20:09:59.610343 (+ 150us) client_negotiation.cc:174] Beginning negotiation
0923 20:09:59.612189 (+ 1846us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:09:59.613758 (+ 1569us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:09:59.613768 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:09:59.615173 (+ 1405us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:09:59.617009 (+ 1836us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:09:59.617060 (+ 51us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:09:59.619773 (+ 2713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:09:59.619778 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:09:59.621081 (+ 1303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:09:59.621097 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:09:59.622875 (+ 1778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:09:59.622884 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:09:59.623430 (+ 546us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:09:59.624406 (+ 976us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:09:59.624438 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:10:00.996902 (+1372464us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.170.190:36595: Server oryx/127.1.170.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":747,"thread_start_us":418,"threads_started":1}
W20260923 20:10:01.000993 10708 heartbeater.cc:644] Failed to heartbeat to 127.1.170.190:36595 (0 consecutive failures): Not authorized: Failed to ping master at 127.1.170.190:36595: Client connection negotiation failed: client connection to 127.1.170.190:36595: Server oryx/127.1.170.190@KRBTEST.COM not found in Kerberos database
Sep 23 20:10:01 dist-test-slave-th4g krb5kdc[9971](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.170.129@KRBTEST.COM for oryx/127.1.170.190@KRBTEST.COM, Server not found in Kerberos database
Sep 23 20:10:01 dist-test-slave-th4g krb5kdc[9971](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.170.129@KRBTEST.COM for oryx/127.1.170.190@KRBTEST.COM, Server not found in Kerberos database
I20260923 20:10:01.020812 10710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:10:01.002006 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.190:36595 (local address 127.1.170.129:52559)
0923 20:10:01.002392 (+ 386us) negotiation.cc:107] Waiting for socket to connect
0923 20:10:01.002420 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0923 20:10:01.002597 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:10:01.003289 (+ 692us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:10:01.003295 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:10:01.003984 (+ 689us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:10:01.004295 (+ 311us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:10:01.004310 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:10:01.006426 (+ 2116us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:10:01.006436 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:10:01.008884 (+ 2448us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:10:01.008904 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:10:01.011110 (+ 2206us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:10:01.011121 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:10:01.011428 (+ 307us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:10:01.012568 (+ 1140us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:10:01.012606 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:10:01.020338 (+ 7732us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.170.190:36595: Server oryx/127.1.170.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":214}
W20260923 20:10:01.021098 10711 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 20:10:01.002092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.170.129:52559 (local address 127.1.170.190:36595)
0923 20:10:01.002351 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0923 20:10:01.002361 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0923 20:10:01.002529 (+ 168us) server_negotiation.cc:407] Connection header received
0923 20:10:01.002778 (+ 249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:10:01.002787 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:10:01.003044 (+ 257us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:10:01.003214 (+ 170us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:10:01.004506 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:10:01.006217 (+ 1711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:10:01.009201 (+ 2984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:10:01.010826 (+ 1625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:10:01.011028 (+ 202us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:10:01.020731 (+ 9703us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.170.129:52559: BlockingRecv error: recv got EOF from 127.1.170.129:52559 (error 108)
Metrics: {"server-negotiator.queue_time_us":96}
Sep 23 20:10:01 dist-test-slave-th4g krb5kdc[9971](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.170.129@KRBTEST.COM for oryx/127.1.170.190@KRBTEST.COM, Server not found in Kerberos database
Sep 23 20:10:01 dist-test-slave-th4g krb5kdc[9971](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.170.129@KRBTEST.COM for oryx/127.1.170.190@KRBTEST.COM, Server not found in Kerberos database
W20260923 20:10:01.040823 10711 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 20:10:01.022896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.170.129:35501 (local address 127.1.170.190:36595)
0923 20:10:01.023121 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0923 20:10:01.023130 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 20:10:01.023157 (+ 27us) server_negotiation.cc:407] Connection header received
0923 20:10:01.023242 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:10:01.023250 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:10:01.023372 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:10:01.023513 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:10:01.024760 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:10:01.026680 (+ 1920us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:10:01.029629 (+ 2949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:10:01.030653 (+ 1024us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:10:01.030827 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:10:01.040384 (+ 9557us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.170.129:35501: BlockingRecv error: recv got EOF from 127.1.170.129:35501 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
I20260923 20:10:01.041801 10710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:10:01.022272 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.190:36595 (local address 127.1.170.129:35501)
0923 20:10:01.022532 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0923 20:10:01.022561 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0923 20:10:01.022719 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:10:01.023604 (+ 885us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:10:01.023612 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:10:01.024229 (+ 617us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:10:01.024550 (+ 321us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:10:01.024566 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:10:01.026952 (+ 2386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:10:01.026962 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:10:01.029350 (+ 2388us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:10:01.029376 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:10:01.031158 (+ 1782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:10:01.031169 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:10:01.031575 (+ 406us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:10:01.032630 (+ 1055us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:10:01.032668 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:10:01.041335 (+ 8667us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.170.190:36595: Server oryx/127.1.170.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":89}
W20260923 20:10:01.042799 10708 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260923 20:10:01.511349 10844 sasl_common.cc:323] Time spent calling sasl_client_start(): real 1.333s user 0.001s sys 0.015s
W20260923 20:10:01.513378 10843 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 20:10:00.168759 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.170.130:49655 (local address 127.1.170.190:36595)
0923 20:10:00.169390 (+ 631us) server_negotiation.cc:206] Beginning negotiation
0923 20:10:00.169401 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0923 20:10:00.170220 (+ 819us) server_negotiation.cc:407] Connection header received
0923 20:10:00.172151 (+ 1931us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:10:00.172157 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:10:00.172285 (+ 128us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:10:00.172412 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:10:00.174289 (+ 1877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:10:00.175054 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:10:00.176614 (+ 1560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:10:00.177040 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:10:00.177176 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:10:01.512994 (+1335818us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.170.130:49655: BlockingRecv error: recv got EOF from 127.1.170.130:49655 (error 108)
Metrics: {"server-negotiator.queue_time_us":448,"thread_start_us":280,"threads_started":1}
I20260923 20:10:01.514583 10844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:10:00.169231 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.190:36595 (local address 127.1.170.130:49655)
0923 20:10:00.170055 (+ 824us) negotiation.cc:107] Waiting for socket to connect
0923 20:10:00.170099 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0923 20:10:00.171856 (+ 1757us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:10:00.172728 (+ 872us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:10:00.172737 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:10:00.173311 (+ 574us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:10:00.174070 (+ 759us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:10:00.174099 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:10:00.175226 (+ 1127us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:10:00.175239 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:10:00.176456 (+ 1217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:10:00.176467 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:10:00.177290 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:10:00.177297 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:10:00.177525 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:10:00.178272 (+ 747us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:10:00.178295 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:10:01.512721 (+1334426us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.170.190:36595: Server oryx/127.1.170.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":529,"thread_start_us":337,"threads_started":1}
W20260923 20:10:01.515609 10841 heartbeater.cc:644] Failed to heartbeat to 127.1.170.190:36595 (0 consecutive failures): Not authorized: Failed to ping master at 127.1.170.190:36595: Client connection negotiation failed: client connection to 127.1.170.190:36595: Server oryx/127.1.170.190@KRBTEST.COM not found in Kerberos database
Sep 23 20:10:01 dist-test-slave-th4g krb5kdc[9971](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.170.130@KRBTEST.COM for oryx/127.1.170.190@KRBTEST.COM, Server not found in Kerberos database
Sep 23 20:10:01 dist-test-slave-th4g krb5kdc[9971](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.170.130@KRBTEST.COM for oryx/127.1.170.190@KRBTEST.COM, Server not found in Kerberos database
W20260923 20:10:01.526654 10843 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 20:10:01.516631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.170.130:34531 (local address 127.1.170.190:36595)
0923 20:10:01.516948 (+ 317us) server_negotiation.cc:206] Beginning negotiation
0923 20:10:01.516953 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 20:10:01.516973 (+ 20us) server_negotiation.cc:407] Connection header received
0923 20:10:01.517333 (+ 360us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:10:01.517339 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:10:01.517419 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:10:01.517526 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:10:01.518420 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:10:01.519187 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:10:01.520433 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:10:01.520856 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:10:01.520993 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:10:01.526418 (+ 5425us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.170.130:34531: BlockingRecv error: recv got EOF from 127.1.170.130:34531 (error 108)
Metrics: {"server-negotiator.queue_time_us":200}
I20260923 20:10:01.526732 10844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:10:01.516439 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.190:36595 (local address 127.1.170.130:34531)
0923 20:10:01.516771 (+ 332us) negotiation.cc:107] Waiting for socket to connect
0923 20:10:01.516827 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0923 20:10:01.517001 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:10:01.517553 (+ 552us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:10:01.517557 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:10:01.518026 (+ 469us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:10:01.518195 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:10:01.518205 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:10:01.519342 (+ 1137us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:10:01.519348 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:10:01.520283 (+ 935us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:10:01.520296 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:10:01.521031 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:10:01.521037 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:10:01.521202 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:10:01.521867 (+ 665us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:10:01.521933 (+ 66us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:10:01.526258 (+ 4325us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.170.190:36595: Server oryx/127.1.170.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":54}
Sep 23 20:10:01 dist-test-slave-th4g krb5kdc[9971](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.170.130@KRBTEST.COM for oryx/127.1.170.190@KRBTEST.COM, Server not found in Kerberos database
Sep 23 20:10:01 dist-test-slave-th4g krb5kdc[9971](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.170.130@KRBTEST.COM for oryx/127.1.170.190@KRBTEST.COM, Server not found in Kerberos database
I20260923 20:10:01.541179 10844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:10:01.527796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.190:36595 (local address 127.1.170.130:58761)
0923 20:10:01.528172 (+ 376us) negotiation.cc:107] Waiting for socket to connect
0923 20:10:01.528215 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0923 20:10:01.528392 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:10:01.528883 (+ 491us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:10:01.528892 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:10:01.529542 (+ 650us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:10:01.529844 (+ 302us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:10:01.529859 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:10:01.530963 (+ 1104us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:10:01.530971 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:10:01.533346 (+ 2375us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:10:01.533371 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:10:01.534305 (+ 934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:10:01.534315 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:10:01.534637 (+ 322us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:10:01.535450 (+ 813us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:10:01.535487 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:10:01.540878 (+ 5391us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.170.190:36595: Server oryx/127.1.170.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":187}
W20260923 20:10:01.541522 10841 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260923 20:10:01.541698 10843 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 20:10:01.528051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.170.130:58761 (local address 127.1.170.190:36595)
0923 20:10:01.528218 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0923 20:10:01.528222 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 20:10:01.528289 (+ 67us) server_negotiation.cc:407] Connection header received
0923 20:10:01.528588 (+ 299us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:10:01.528598 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:10:01.528670 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:10:01.528789 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:10:01.530048 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:10:01.530793 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:10:01.533555 (+ 2762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:10:01.534108 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:10:01.534211 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:10:01.541076 (+ 6865us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.170.130:58761: BlockingRecv error: recv got EOF from 127.1.170.130:58761 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Sep 23 20:10:02 dist-test-slave-th4g krb5kdc[9971](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.170.129@KRBTEST.COM for oryx/127.1.170.190@KRBTEST.COM, Server not found in Kerberos database
Sep 23 20:10:02 dist-test-slave-th4g krb5kdc[9971](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.170.129@KRBTEST.COM for oryx/127.1.170.190@KRBTEST.COM, Server not found in Kerberos database
W20260923 20:10:02.056571 10979 sasl_common.cc:323] Time spent calling sasl_client_start(): real 1.153s user 0.004s sys 0.012s
I20260923 20:10:02.057291 10982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:10:02.043472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.190:36595 (local address 127.1.170.129:48689)
0923 20:10:02.046691 (+ 3219us) negotiation.cc:107] Waiting for socket to connect
0923 20:10:02.046714 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0923 20:10:02.046868 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:10:02.047703 (+ 835us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:10:02.047712 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:10:02.048256 (+ 544us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:10:02.048460 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:10:02.048472 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:10:02.049749 (+ 1277us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:10:02.049755 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:10:02.050842 (+ 1087us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:10:02.050855 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:10:02.051628 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:10:02.051634 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:10:02.051815 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:10:02.052706 (+ 891us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:10:02.052726 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:10:02.056934 (+ 4208us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.170.190:36595: Server oryx/127.1.170.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":3087,"thread_start_us":2963,"threads_started":1}
W20260923 20:10:02.057709 10980 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 20:10:00.877660 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.170.131:49071 (local address 127.1.170.190:36595)
0923 20:10:00.890421 (+ 12761us) server_negotiation.cc:206] Beginning negotiation
0923 20:10:00.890429 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 20:10:00.894218 (+ 3789us) server_negotiation.cc:407] Connection header received
0923 20:10:00.894293 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:10:00.894300 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:10:00.894394 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:10:00.894500 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:10:00.898148 (+ 3648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:10:00.899319 (+ 1171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:10:00.901662 (+ 2343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:10:00.902283 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:10:00.902351 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:10:02.057473 (+1155122us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.170.131:49071: BlockingRecv error: recv got EOF from 127.1.170.131:49071 (error 108)
Metrics: {"server-negotiator.queue_time_us":12620,"thread_start_us":12505,"threads_started":1}
W20260923 20:10:02.057945 10983 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 20:10:02.043544 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.170.129:48689 (local address 127.1.170.190:36595)
0923 20:10:02.047302 (+ 3758us) server_negotiation.cc:206] Beginning negotiation
0923 20:10:02.047308 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:10:02.047327 (+ 19us) server_negotiation.cc:407] Connection header received
0923 20:10:02.047386 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:10:02.047391 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:10:02.047490 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:10:02.047648 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:10:02.048614 (+ 966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:10:02.049513 (+ 899us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:10:02.051010 (+ 1497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:10:02.051430 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:10:02.051582 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:10:02.057198 (+ 5616us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.170.129:48689: BlockingRecv error: recv got EOF from 127.1.170.129:48689 (error 108)
Metrics: {"server-negotiator.queue_time_us":3626,"thread_start_us":3523,"threads_started":1}
I20260923 20:10:02.058477 10979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:10:00.876845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.190:36595 (local address 127.1.170.131:49071)
0923 20:10:00.891078 (+ 14233us) negotiation.cc:107] Waiting for socket to connect
0923 20:10:00.891134 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0923 20:10:00.893775 (+ 2641us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:10:00.895627 (+ 1852us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:10:00.895647 (+ 20us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:10:00.896727 (+ 1080us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:10:00.897915 (+ 1188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:10:00.897953 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:10:00.899501 (+ 1548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:10:00.899507 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:10:00.901501 (+ 1994us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:10:00.901514 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:10:00.902744 (+ 1230us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:10:00.902751 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:10:00.903096 (+ 345us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:10:00.903831 (+ 735us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:10:00.903871 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:10:02.057330 (+1153459us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.170.190:36595: Server oryx/127.1.170.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":13830,"thread_start_us":13539,"threads_started":1}
W20260923 20:10:02.060248 10975 heartbeater.cc:644] Failed to heartbeat to 127.1.170.190:36595 (0 consecutive failures): Not authorized: Failed to ping master at 127.1.170.190:36595: Client connection negotiation failed: client connection to 127.1.170.190:36595: Server oryx/127.1.170.190@KRBTEST.COM not found in Kerberos database
Sep 23 20:10:02 dist-test-slave-th4g krb5kdc[9971](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.170.131@KRBTEST.COM for oryx/127.1.170.190@KRBTEST.COM, Server not found in Kerberos database
Sep 23 20:10:02 dist-test-slave-th4g krb5kdc[9971](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.170.131@KRBTEST.COM for oryx/127.1.170.190@KRBTEST.COM, Server not found in Kerberos database
I20260923 20:10:02.072834 10979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:10:02.061031 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.190:36595 (local address 127.1.170.131:33143)
0923 20:10:02.061338 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0923 20:10:02.061355 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0923 20:10:02.061468 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:10:02.061981 (+ 513us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:10:02.061986 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:10:02.062432 (+ 446us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:10:02.062610 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:10:02.062619 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:10:02.064031 (+ 1412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:10:02.064036 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:10:02.065164 (+ 1128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:10:02.065179 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:10:02.066415 (+ 1236us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:10:02.066420 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:10:02.066547 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:10:02.067321 (+ 774us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:10:02.067350 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:10:02.072490 (+ 5140us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.170.190:36595: Server oryx/127.1.170.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":96}
W20260923 20:10:02.073069 10983 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 20:10:02.061087 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.170.131:33143 (local address 127.1.170.190:36595)
0923 20:10:02.061345 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0923 20:10:02.061351 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:10:02.061437 (+ 86us) server_negotiation.cc:407] Connection header received
0923 20:10:02.061588 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:10:02.061592 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:10:02.061699 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:10:02.061820 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:10:02.062792 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:10:02.063835 (+ 1043us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:10:02.065378 (+ 1543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:10:02.066207 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:10:02.066323 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:10:02.072839 (+ 6516us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.170.131:33143: BlockingRecv error: recv got EOF from 127.1.170.131:33143 (error 108)
Metrics: {"server-negotiator.queue_time_us":146}
Sep 23 20:10:02 dist-test-slave-th4g krb5kdc[9971](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.170.131@KRBTEST.COM for oryx/127.1.170.190@KRBTEST.COM, Server not found in Kerberos database
Sep 23 20:10:02 dist-test-slave-th4g krb5kdc[9971](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.170.131@KRBTEST.COM for oryx/127.1.170.190@KRBTEST.COM, Server not found in Kerberos database
I20260923 20:10:02.083968 10979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:10:02.073730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.190:36595 (local address 127.1.170.131:55735)
0923 20:10:02.074019 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0923 20:10:02.074039 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0923 20:10:02.074165 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:10:02.074607 (+ 442us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:10:02.074611 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:10:02.075107 (+ 496us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:10:02.075337 (+ 230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:10:02.075349 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:10:02.076444 (+ 1095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:10:02.076448 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:10:02.077551 (+ 1103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:10:02.077564 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:10:02.078706 (+ 1142us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:10:02.078715 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:10:02.078866 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:10:02.079642 (+ 776us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:10:02.079666 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:10:02.083756 (+ 4090us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.170.190:36595: Server oryx/127.1.170.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":176}
W20260923 20:10:02.084415 10975 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260923 20:10:02.084641 10983 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 20:10:02.073896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.170.131:55735 (local address 127.1.170.190:36595)
0923 20:10:02.074071 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0923 20:10:02.074076 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 20:10:02.074096 (+ 20us) server_negotiation.cc:407] Connection header received
0923 20:10:02.074316 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:10:02.074321 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:10:02.074438 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:10:02.074543 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:10:02.075538 (+ 995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:10:02.076296 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:10:02.077731 (+ 1435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:10:02.078431 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:10:02.078560 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:10:02.084364 (+ 5804us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.170.131:55735: BlockingRecv error: recv got EOF from 127.1.170.131:55735 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
Sep 23 20:10:02 dist-test-slave-th4g krb5kdc[9971](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.170.130@KRBTEST.COM for oryx/127.1.170.190@KRBTEST.COM, Server not found in Kerberos database
Sep 23 20:10:02 dist-test-slave-th4g krb5kdc[9971](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.170.130@KRBTEST.COM for oryx/127.1.170.190@KRBTEST.COM, Server not found in Kerberos database
I20260923 20:10:02.557391 10984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:10:02.542355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.190:36595 (local address 127.1.170.130:58729)
0923 20:10:02.543111 (+ 756us) negotiation.cc:107] Waiting for socket to connect
0923 20:10:02.543128 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0923 20:10:02.543351 (+ 223us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:10:02.543712 (+ 361us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:10:02.543717 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:10:02.544148 (+ 431us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:10:02.544287 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:10:02.544296 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:10:02.545286 (+ 990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:10:02.545291 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:10:02.546342 (+ 1051us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:10:02.546354 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:10:02.547100 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:10:02.547106 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:10:02.547266 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:10:02.552661 (+ 5395us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:10:02.552682 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:10:02.557030 (+ 4348us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.170.190:36595: Server oryx/127.1.170.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":655,"thread_start_us":417,"threads_started":1}
W20260923 20:10:02.557492 10983 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 20:10:02.542551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.170.130:58729 (local address 127.1.170.190:36595)
0923 20:10:02.542858 (+ 307us) server_negotiation.cc:206] Beginning negotiation
0923 20:10:02.542866 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 20:10:02.543242 (+ 376us) server_negotiation.cc:407] Connection header received
0923 20:10:02.543507 (+ 265us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:10:02.543512 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:10:02.543578 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:10:02.543686 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:10:02.544440 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:10:02.545142 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:10:02.546491 (+ 1349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:10:02.546903 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:10:02.547037 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:10:02.557258 (+ 10221us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.170.130:58729: BlockingRecv error: recv got EOF from 127.1.170.130:58729 (error 108)
Metrics: {"server-negotiator.queue_time_us":103}
Sep 23 20:10:03 dist-test-slave-th4g krb5kdc[9971](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.170.129@KRBTEST.COM for oryx/127.1.170.190@KRBTEST.COM, Server not found in Kerberos database
Sep 23 20:10:03 dist-test-slave-th4g krb5kdc[9971](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.170.129@KRBTEST.COM for oryx/127.1.170.190@KRBTEST.COM, Server not found in Kerberos database
I20260923 20:10:03.071856 10985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:10:03.058528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.190:36595 (local address 127.1.170.129:43001)
0923 20:10:03.059137 (+ 609us) negotiation.cc:107] Waiting for socket to connect
0923 20:10:03.059154 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0923 20:10:03.059335 (+ 181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:10:03.060306 (+ 971us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:10:03.060313 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:10:03.060953 (+ 640us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:10:03.061296 (+ 343us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:10:03.061314 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:10:03.062862 (+ 1548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:10:03.062868 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:10:03.063879 (+ 1011us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:10:03.063897 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:10:03.065072 (+ 1175us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:10:03.065077 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:10:03.065222 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:10:03.065998 (+ 776us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:10:03.066017 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:10:03.071345 (+ 5328us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.170.190:36595: Server oryx/127.1.170.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":501,"thread_start_us":346,"threads_started":1}
W20260923 20:10:03.071956 10986 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 20:10:03.058705 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.170.129:43001 (local address 127.1.170.190:36595)
0923 20:10:03.059899 (+ 1194us) server_negotiation.cc:206] Beginning negotiation
0923 20:10:03.059906 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:10:03.059929 (+ 23us) server_negotiation.cc:407] Connection header received
0923 20:10:03.059997 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:10:03.060000 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:10:03.060066 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:10:03.060177 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:10:03.061565 (+ 1388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:10:03.062610 (+ 1045us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:10:03.064247 (+ 1637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:10:03.064832 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:10:03.064948 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:10:03.071701 (+ 6753us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.170.129:43001: BlockingRecv error: recv got EOF from 127.1.170.129:43001 (error 108)
Metrics: {"server-negotiator.queue_time_us":747,"thread_start_us":657,"threads_started":1}
Sep 23 20:10:03 dist-test-slave-th4g krb5kdc[9971](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.170.131@KRBTEST.COM for oryx/127.1.170.190@KRBTEST.COM, Server not found in Kerberos database
Sep 23 20:10:03 dist-test-slave-th4g krb5kdc[9971](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.170.131@KRBTEST.COM for oryx/127.1.170.190@KRBTEST.COM, Server not found in Kerberos database
I20260923 20:10:03.095412 10987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:10:03.085610 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.190:36595 (local address 127.1.170.131:57301)
0923 20:10:03.086252 (+ 642us) negotiation.cc:107] Waiting for socket to connect
0923 20:10:03.086270 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0923 20:10:03.086455 (+ 185us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:10:03.086866 (+ 411us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:10:03.086871 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:10:03.087342 (+ 471us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:10:03.087549 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:10:03.087562 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:10:03.088718 (+ 1156us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:10:03.088723 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:10:03.089686 (+ 963us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:10:03.089698 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:10:03.090681 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:10:03.090686 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:10:03.090818 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:10:03.091458 (+ 640us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:10:03.091476 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:10:03.095175 (+ 3699us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.170.190:36595: Server oryx/127.1.170.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":531,"thread_start_us":516,"threads_started":1}
W20260923 20:10:03.095773 10986 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 20:10:03.085804 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.170.131:57301 (local address 127.1.170.190:36595)
0923 20:10:03.086094 (+ 290us) server_negotiation.cc:206] Beginning negotiation
0923 20:10:03.086102 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 20:10:03.086583 (+ 481us) server_negotiation.cc:407] Connection header received
0923 20:10:03.086646 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:10:03.086650 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:10:03.086711 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:10:03.086828 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:10:03.087763 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:10:03.088556 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:10:03.089826 (+ 1270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:10:03.090314 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:10:03.090413 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:10:03.095514 (+ 5101us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.170.131:57301: BlockingRecv error: recv got EOF from 127.1.170.131:57301 (error 108)
Metrics: {"server-negotiator.queue_time_us":113}
Sep 23 20:10:03 dist-test-slave-th4g krb5kdc[9971](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.170.130@KRBTEST.COM for oryx/127.1.170.190@KRBTEST.COM, Server not found in Kerberos database
Sep 23 20:10:03 dist-test-slave-th4g krb5kdc[9971](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.170.130@KRBTEST.COM for oryx/127.1.170.190@KRBTEST.COM, Server not found in Kerberos database
I20260923 20:10:03.569585 10988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:10:03.558730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.190:36595 (local address 127.1.170.130:36149)
0923 20:10:03.559388 (+ 658us) negotiation.cc:107] Waiting for socket to connect
0923 20:10:03.559404 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0923 20:10:03.559582 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:10:03.559955 (+ 373us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:10:03.559959 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:10:03.560377 (+ 418us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:10:03.560551 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:10:03.560560 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:10:03.561623 (+ 1063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:10:03.561630 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:10:03.562772 (+ 1142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:10:03.562799 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:10:03.563751 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:10:03.563757 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:10:03.563953 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:10:03.564633 (+ 680us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:10:03.564651 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:10:03.569175 (+ 4524us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.170.190:36595: Server oryx/127.1.170.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":527,"thread_start_us":389,"threads_started":1}
W20260923 20:10:03.570070 10986 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 20:10:03.558871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.170.130:36149 (local address 127.1.170.190:36595)
0923 20:10:03.559246 (+ 375us) server_negotiation.cc:206] Beginning negotiation
0923 20:10:03.559253 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:10:03.559493 (+ 240us) server_negotiation.cc:407] Connection header received
0923 20:10:03.559748 (+ 255us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:10:03.559752 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:10:03.559815 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:10:03.559928 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:10:03.560698 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:10:03.561477 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:10:03.563113 (+ 1636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:10:03.563559 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:10:03.563724 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:10:03.569484 (+ 5760us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.170.130:36149: BlockingRecv error: recv got EOF from 127.1.170.130:36149 (error 108)
Metrics: {"server-negotiator.queue_time_us":206}
Sep 23 20:10:04 dist-test-slave-th4g krb5kdc[9971](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.170.129@KRBTEST.COM for oryx/127.1.170.190@KRBTEST.COM, Server not found in Kerberos database
Sep 23 20:10:04 dist-test-slave-th4g krb5kdc[9971](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.170.129@KRBTEST.COM for oryx/127.1.170.190@KRBTEST.COM, Server not found in Kerberos database
I20260923 20:10:04.084713 10989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:10:04.073237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.190:36595 (local address 127.1.170.129:55741)
0923 20:10:04.073790 (+ 553us) negotiation.cc:107] Waiting for socket to connect
0923 20:10:04.073808 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0923 20:10:04.073981 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:10:04.074291 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:10:04.074294 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:10:04.074624 (+ 330us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:10:04.074820 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:10:04.074829 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:10:04.075873 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:10:04.075877 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:10:04.077096 (+ 1219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:10:04.077108 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:10:04.078304 (+ 1196us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:10:04.078314 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:10:04.078668 (+ 354us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:10:04.079867 (+ 1199us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:10:04.079888 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:10:04.084460 (+ 4572us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.170.190:36595: Server oryx/127.1.170.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":426,"thread_start_us":294,"threads_started":1}
W20260923 20:10:04.085180 10990 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 20:10:04.073447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.170.129:55741 (local address 127.1.170.190:36595)
0923 20:10:04.073969 (+ 522us) server_negotiation.cc:206] Beginning negotiation
0923 20:10:04.073973 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 20:10:04.073991 (+ 18us) server_negotiation.cc:407] Connection header received
0923 20:10:04.074075 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:10:04.074079 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:10:04.074143 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:10:04.074241 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:10:04.075003 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:10:04.075705 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:10:04.077308 (+ 1603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:10:04.078054 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:10:04.078157 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:10:04.084805 (+ 6648us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.170.129:55741: BlockingRecv error: recv got EOF from 127.1.170.129:55741 (error 108)
Metrics: {"server-negotiator.queue_time_us":346,"thread_start_us":255,"threads_started":1}
Sep 23 20:10:04 dist-test-slave-th4g krb5kdc[9971](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.170.131@KRBTEST.COM for oryx/127.1.170.190@KRBTEST.COM, Server not found in Kerberos database
Sep 23 20:10:04 dist-test-slave-th4g krb5kdc[9971](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.170.131@KRBTEST.COM for oryx/127.1.170.190@KRBTEST.COM, Server not found in Kerberos database
I20260923 20:10:04.108341 10991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:10:04.096520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.190:36595 (local address 127.1.170.131:35683)
0923 20:10:04.097249 (+ 729us) negotiation.cc:107] Waiting for socket to connect
0923 20:10:04.097268 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0923 20:10:04.097448 (+ 180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:10:04.098094 (+ 646us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:10:04.098099 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:10:04.098625 (+ 526us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:10:04.098845 (+ 220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:10:04.098856 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:10:04.100061 (+ 1205us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:10:04.100065 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:10:04.101196 (+ 1131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:10:04.101220 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:10:04.102094 (+ 874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:10:04.102101 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:10:04.102301 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:10:04.103042 (+ 741us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:10:04.103068 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:10:04.108052 (+ 4984us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.170.190:36595: Server oryx/127.1.170.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":619,"thread_start_us":378,"threads_started":1}
W20260923 20:10:04.108893 10990 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 20:10:04.096825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.170.131:35683 (local address 127.1.170.190:36595)
0923 20:10:04.097125 (+ 300us) server_negotiation.cc:206] Beginning negotiation
0923 20:10:04.097132 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:10:04.097458 (+ 326us) server_negotiation.cc:407] Connection header received
0923 20:10:04.097620 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:10:04.097624 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:10:04.097697 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:10:04.097798 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:10:04.099075 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:10:04.099892 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:10:04.101380 (+ 1488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:10:04.101798 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:10:04.102035 (+ 237us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:10:04.108662 (+ 6627us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.170.131:35683: BlockingRecv error: recv got EOF from 127.1.170.131:35683 (error 108)
Metrics: {"server-negotiator.queue_time_us":92}
Sep 23 20:10:04 dist-test-slave-th4g krb5kdc[9971](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.170.130@KRBTEST.COM for oryx/127.1.170.190@KRBTEST.COM, Server not found in Kerberos database
Sep 23 20:10:04 dist-test-slave-th4g krb5kdc[9971](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.170.130@KRBTEST.COM for oryx/127.1.170.190@KRBTEST.COM, Server not found in Kerberos database
I20260923 20:10:04.580571 10992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:10:04.570905 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.190:36595 (local address 127.1.170.130:46609)
0923 20:10:04.571479 (+ 574us) negotiation.cc:107] Waiting for socket to connect
0923 20:10:04.571495 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0923 20:10:04.571660 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:10:04.572008 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:10:04.572013 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:10:04.572417 (+ 404us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:10:04.572617 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:10:04.572627 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:10:04.573682 (+ 1055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:10:04.573688 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:10:04.574618 (+ 930us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:10:04.574628 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:10:04.575339 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:10:04.575345 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:10:04.575531 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:10:04.576166 (+ 635us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:10:04.576183 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:10:04.580141 (+ 3958us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.170.190:36595: Server oryx/127.1.170.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":454,"thread_start_us":311,"threads_started":1}
W20260923 20:10:04.580598 10990 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 20:10:04.571069 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.170.130:46609 (local address 127.1.170.190:36595)
0923 20:10:04.571319 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0923 20:10:04.571325 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:10:04.571581 (+ 256us) server_negotiation.cc:407] Connection header received
0923 20:10:04.571788 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:10:04.571798 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:10:04.571875 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:10:04.571983 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:10:04.572790 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:10:04.573526 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:10:04.574760 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:10:04.575188 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:10:04.575327 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:10:04.580359 (+ 5032us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.170.130:46609: BlockingRecv error: recv got EOF from 127.1.170.130:46609 (error 108)
Metrics: {"server-negotiator.queue_time_us":83}
Sep 23 20:10:05 dist-test-slave-th4g krb5kdc[9971](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.170.129@KRBTEST.COM for oryx/127.1.170.190@KRBTEST.COM, Server not found in Kerberos database
Sep 23 20:10:05 dist-test-slave-th4g krb5kdc[9971](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.170.129@KRBTEST.COM for oryx/127.1.170.190@KRBTEST.COM, Server not found in Kerberos database
I20260923 20:10:05.095046 10993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:10:05.085841 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.190:36595 (local address 127.1.170.129:56711)
0923 20:10:05.086519 (+ 678us) negotiation.cc:107] Waiting for socket to connect
0923 20:10:05.086548 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0923 20:10:05.086692 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:10:05.087108 (+ 416us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:10:05.087113 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:10:05.087607 (+ 494us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:10:05.087780 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:10:05.087789 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:10:05.088815 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:10:05.088822 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:10:05.089788 (+ 966us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:10:05.089798 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:10:05.090489 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:10:05.090494 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:10:05.090669 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:10:05.091296 (+ 627us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:10:05.091313 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:10:05.094798 (+ 3485us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.170.190:36595: Server oryx/127.1.170.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":515,"thread_start_us":316,"threads_started":1}
W20260923 20:10:05.095312 10994 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 20:10:05.085970 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.170.129:56711 (local address 127.1.170.190:36595)
0923 20:10:05.086813 (+ 843us) server_negotiation.cc:206] Beginning negotiation
0923 20:10:05.086817 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 20:10:05.086833 (+ 16us) server_negotiation.cc:407] Connection header received
0923 20:10:05.086879 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:10:05.086882 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:10:05.086953 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:10:05.087078 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:10:05.087915 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:10:05.088694 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:10:05.089945 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:10:05.090344 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:10:05.090480 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:10:05.095033 (+ 4553us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.170.129:56711: BlockingRecv error: recv got EOF from 127.1.170.129:56711 (error 108)
Metrics: {"server-negotiator.queue_time_us":726,"thread_start_us":625,"threads_started":1}
Sep 23 20:10:05 dist-test-slave-th4g krb5kdc[9971](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.170.131@KRBTEST.COM for oryx/127.1.170.190@KRBTEST.COM, Server not found in Kerberos database
Sep 23 20:10:05 dist-test-slave-th4g krb5kdc[9971](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.170.131@KRBTEST.COM for oryx/127.1.170.190@KRBTEST.COM, Server not found in Kerberos database
I20260923 20:10:05.118422 10995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:10:05.109436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.190:36595 (local address 127.1.170.131:59187)
0923 20:10:05.109994 (+ 558us) negotiation.cc:107] Waiting for socket to connect
0923 20:10:05.110009 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0923 20:10:05.110127 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:10:05.110424 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:10:05.110427 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:10:05.110892 (+ 465us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:10:05.111073 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:10:05.111081 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:10:05.112267 (+ 1186us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:10:05.112272 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:10:05.113316 (+ 1044us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:10:05.113328 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:10:05.114042 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:10:05.114046 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:10:05.114172 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:10:05.114715 (+ 543us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:10:05.114732 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:10:05.118184 (+ 3452us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.170.190:36595: Server oryx/127.1.170.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":423,"thread_start_us":291,"threads_started":1}
W20260923 20:10:05.118611 10994 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 20:10:05.109558 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.170.131:59187 (local address 127.1.170.190:36595)
0923 20:10:05.109721 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0923 20:10:05.109726 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 20:10:05.110083 (+ 357us) server_negotiation.cc:407] Connection header received
0923 20:10:05.110224 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:10:05.110229 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:10:05.110290 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:10:05.110386 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:10:05.111183 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:10:05.112114 (+ 931us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:10:05.113444 (+ 1330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:10:05.113852 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:10:05.114003 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:10:05.118401 (+ 4398us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.170.131:59187: BlockingRecv error: recv got EOF from 127.1.170.131:59187 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Sep 23 20:10:05 dist-test-slave-th4g krb5kdc[9971](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.170.130@KRBTEST.COM for oryx/127.1.170.190@KRBTEST.COM, Server not found in Kerberos database
Sep 23 20:10:05 dist-test-slave-th4g krb5kdc[9971](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.170.130@KRBTEST.COM for oryx/127.1.170.190@KRBTEST.COM, Server not found in Kerberos database
I20260923 20:10:05.591262 10996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:10:05.581795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.190:36595 (local address 127.1.170.130:37127)
0923 20:10:05.582516 (+ 721us) negotiation.cc:107] Waiting for socket to connect
0923 20:10:05.582531 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0923 20:10:05.582644 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:10:05.582964 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:10:05.582967 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:10:05.583314 (+ 347us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:10:05.583466 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:10:05.583481 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:10:05.584522 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:10:05.584526 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:10:05.585827 (+ 1301us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:10:05.585844 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:10:05.586733 (+ 889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:10:05.586737 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:10:05.586933 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:10:05.587652 (+ 719us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:10:05.587670 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:10:05.591018 (+ 3348us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.170.190:36595: Server oryx/127.1.170.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":616,"thread_start_us":512,"threads_started":1}
W20260923 20:10:05.591420 10994 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 20:10:05.581845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.170.130:37127 (local address 127.1.170.190:36595)
0923 20:10:05.582129 (+ 284us) server_negotiation.cc:206] Beginning negotiation
0923 20:10:05.582135 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:10:05.582630 (+ 495us) server_negotiation.cc:407] Connection header received
0923 20:10:05.582753 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:10:05.582757 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:10:05.582831 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:10:05.582920 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:10:05.583621 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:10:05.584400 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:10:05.585978 (+ 1578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:10:05.586568 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:10:05.586709 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:10:05.591193 (+ 4484us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.170.130:37127: BlockingRecv error: recv got EOF from 127.1.170.130:37127 (error 108)
Metrics: {"server-negotiator.queue_time_us":119}
Sep 23 20:10:06 dist-test-slave-th4g krb5kdc[9971](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.170.129@KRBTEST.COM for oryx/127.1.170.190@KRBTEST.COM, Server not found in Kerberos database
Sep 23 20:10:06 dist-test-slave-th4g krb5kdc[9971](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.170.129@KRBTEST.COM for oryx/127.1.170.190@KRBTEST.COM, Server not found in Kerberos database
I20260923 20:10:06.105818 10997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:10:06.096067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.190:36595 (local address 127.1.170.129:38277)
0923 20:10:06.096581 (+ 514us) negotiation.cc:107] Waiting for socket to connect
0923 20:10:06.096599 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0923 20:10:06.096714 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:10:06.097176 (+ 462us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:10:06.097180 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:10:06.097639 (+ 459us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:10:06.097790 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:10:06.097799 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:10:06.098850 (+ 1051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:10:06.098854 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:10:06.099898 (+ 1044us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:10:06.099909 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:10:06.100721 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:10:06.100726 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:10:06.100896 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:10:06.101634 (+ 738us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:10:06.101670 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:10:06.105592 (+ 3922us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.170.190:36595: Server oryx/127.1.170.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":367,"thread_start_us":249,"threads_started":1}
W20260923 20:10:06.106168 10998 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 20:10:06.096208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.170.129:38277 (local address 127.1.170.190:36595)
0923 20:10:06.096840 (+ 632us) server_negotiation.cc:206] Beginning negotiation
0923 20:10:06.096844 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 20:10:06.096860 (+ 16us) server_negotiation.cc:407] Connection header received
0923 20:10:06.096907 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:10:06.096911 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:10:06.096975 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:10:06.097064 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:10:06.097967 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:10:06.098725 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:10:06.100065 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:10:06.100566 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:10:06.100696 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:10:06.105849 (+ 5153us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.170.129:38277: BlockingRecv error: recv got EOF from 127.1.170.129:38277 (error 108)
Metrics: {"server-negotiator.queue_time_us":535,"thread_start_us":427,"threads_started":1}
Sep 23 20:10:06 dist-test-slave-th4g krb5kdc[9971](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.170.131@KRBTEST.COM for oryx/127.1.170.190@KRBTEST.COM, Server not found in Kerberos database
Sep 23 20:10:06 dist-test-slave-th4g krb5kdc[9971](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.170.131@KRBTEST.COM for oryx/127.1.170.190@KRBTEST.COM, Server not found in Kerberos database
I20260923 20:10:06.128656 10999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:10:06.119382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.190:36595 (local address 127.1.170.131:51433)
0923 20:10:06.119901 (+ 519us) negotiation.cc:107] Waiting for socket to connect
0923 20:10:06.119917 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0923 20:10:06.120021 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:10:06.120339 (+ 318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:10:06.120342 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:10:06.120688 (+ 346us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:10:06.120853 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:10:06.120862 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:10:06.121841 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:10:06.121845 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:10:06.122901 (+ 1056us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:10:06.122912 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:10:06.123576 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:10:06.123580 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:10:06.123712 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:10:06.124325 (+ 613us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:10:06.124343 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:10:06.128406 (+ 4063us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.170.190:36595: Server oryx/127.1.170.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":410,"thread_start_us":279,"threads_started":1}
W20260923 20:10:06.128854 10998 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 20:10:06.119441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.170.131:51433 (local address 127.1.170.190:36595)
0923 20:10:06.119603 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0923 20:10:06.119607 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 20:10:06.119992 (+ 385us) server_negotiation.cc:407] Connection header received
0923 20:10:06.120113 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:10:06.120117 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:10:06.120177 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:10:06.120265 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:10:06.120996 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:10:06.121717 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:10:06.123027 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:10:06.123418 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:10:06.123516 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:10:06.128611 (+ 5095us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.170.131:51433: BlockingRecv error: recv got EOF from 127.1.170.131:51433 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Sep 23 20:10:06 dist-test-slave-th4g krb5kdc[9971](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.170.130@KRBTEST.COM for oryx/127.1.170.190@KRBTEST.COM, Server not found in Kerberos database
Sep 23 20:10:06 dist-test-slave-th4g krb5kdc[9971](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.170.130@KRBTEST.COM for oryx/127.1.170.190@KRBTEST.COM, Server not found in Kerberos database
I20260923 20:10:06.601200 11000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:10:06.592208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.190:36595 (local address 127.1.170.130:41471)
0923 20:10:06.592737 (+ 529us) negotiation.cc:107] Waiting for socket to connect
0923 20:10:06.592753 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0923 20:10:06.592859 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:10:06.593202 (+ 343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:10:06.593205 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:10:06.593642 (+ 437us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:10:06.593790 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:10:06.593798 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:10:06.594812 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:10:06.594816 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:10:06.595710 (+ 894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:10:06.595721 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:10:06.596436 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:10:06.596440 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:10:06.596577 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:10:06.597252 (+ 675us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:10:06.597270 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:10:06.600875 (+ 3605us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.170.190:36595: Server oryx/127.1.170.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":399,"thread_start_us":282,"threads_started":1}
W20260923 20:10:06.601305 10998 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 20:10:06.592380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.170.130:41471 (local address 127.1.170.190:36595)
0923 20:10:06.592568 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0923 20:10:06.592573 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 20:10:06.592834 (+ 261us) server_negotiation.cc:407] Connection header received
0923 20:10:06.592955 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:10:06.592958 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:10:06.593029 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:10:06.593126 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:10:06.593972 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:10:06.594686 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:10:06.595843 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:10:06.596262 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:10:06.596417 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:10:06.601056 (+ 4639us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.170.130:41471: BlockingRecv error: recv got EOF from 127.1.170.130:41471 (error 108)
Metrics: {"server-negotiator.queue_time_us":81}
Sep 23 20:10:07 dist-test-slave-th4g krb5kdc[9971](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.170.129@KRBTEST.COM for oryx/127.1.170.190@KRBTEST.COM, Server not found in Kerberos database
Sep 23 20:10:07 dist-test-slave-th4g krb5kdc[9971](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.170.129@KRBTEST.COM for oryx/127.1.170.190@KRBTEST.COM, Server not found in Kerberos database
I20260923 20:10:07.118419 11001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:10:07.106899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.190:36595 (local address 127.1.170.129:46225)
0923 20:10:07.107579 (+ 680us) negotiation.cc:107] Waiting for socket to connect
0923 20:10:07.107605 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0923 20:10:07.107749 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:10:07.108552 (+ 803us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:10:07.108556 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:10:07.108931 (+ 375us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:10:07.109087 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:10:07.109128 (+ 41us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:10:07.110257 (+ 1129us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:10:07.110261 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:10:07.111690 (+ 1429us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:10:07.111705 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:10:07.112802 (+ 1097us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:10:07.112809 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:10:07.113005 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:10:07.113850 (+ 845us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:10:07.114036 (+ 186us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:10:07.118192 (+ 4156us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.170.190:36595: Server oryx/127.1.170.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":490,"thread_start_us":326,"threads_started":1}
W20260923 20:10:07.118764 11002 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 20:10:07.106966 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.170.129:46225 (local address 127.1.170.190:36595)
0923 20:10:07.107659 (+ 693us) server_negotiation.cc:206] Beginning negotiation
0923 20:10:07.107664 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 20:10:07.107686 (+ 22us) server_negotiation.cc:407] Connection header received
0923 20:10:07.107938 (+ 252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:10:07.107943 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:10:07.108361 (+ 418us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:10:07.108471 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:10:07.109309 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:10:07.110072 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:10:07.111830 (+ 1758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:10:07.112430 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:10:07.112584 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:10:07.118459 (+ 5875us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.170.129:46225: BlockingRecv error: recv got EOF from 127.1.170.129:46225 (error 108)
Metrics: {"server-negotiator.queue_time_us":447,"thread_start_us":338,"threads_started":1}
Sep 23 20:10:07 dist-test-slave-th4g krb5kdc[9971](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.170.131@KRBTEST.COM for oryx/127.1.170.190@KRBTEST.COM, Server not found in Kerberos database
Sep 23 20:10:07 dist-test-slave-th4g krb5kdc[9971](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.170.131@KRBTEST.COM for oryx/127.1.170.190@KRBTEST.COM, Server not found in Kerberos database
I20260923 20:10:07.139060 11003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:10:07.129822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.190:36595 (local address 127.1.170.131:57401)
0923 20:10:07.130444 (+ 622us) negotiation.cc:107] Waiting for socket to connect
0923 20:10:07.130461 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0923 20:10:07.130583 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:10:07.130912 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:10:07.130915 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:10:07.131361 (+ 446us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:10:07.131508 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:10:07.131516 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:10:07.132577 (+ 1061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:10:07.132582 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:10:07.133551 (+ 969us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:10:07.133562 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:10:07.134378 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:10:07.134382 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:10:07.134518 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:10:07.135122 (+ 604us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:10:07.135166 (+ 44us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:10:07.138729 (+ 3563us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.170.190:36595: Server oryx/127.1.170.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":526,"thread_start_us":396,"threads_started":1}
W20260923 20:10:07.139119 11002 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 20:10:07.129955 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.170.131:57401 (local address 127.1.170.190:36595)
0923 20:10:07.130216 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0923 20:10:07.130225 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 20:10:07.130540 (+ 315us) server_negotiation.cc:407] Connection header received
0923 20:10:07.130717 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:10:07.130721 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:10:07.130786 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:10:07.130875 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:10:07.131654 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:10:07.132428 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:10:07.133713 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:10:07.134236 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:10:07.134341 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:10:07.138923 (+ 4582us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.170.131:57401: BlockingRecv error: recv got EOF from 127.1.170.131:57401 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Sep 23 20:10:07 dist-test-slave-th4g krb5kdc[9971](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.170.130@KRBTEST.COM for oryx/127.1.170.190@KRBTEST.COM, Server not found in Kerberos database
Sep 23 20:10:07 dist-test-slave-th4g krb5kdc[9971](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.170.130@KRBTEST.COM for oryx/127.1.170.190@KRBTEST.COM, Server not found in Kerberos database
I20260923 20:10:07.619546 11004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:10:07.602357 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.190:36595 (local address 127.1.170.130:39129)
0923 20:10:07.602853 (+ 496us) negotiation.cc:107] Waiting for socket to connect
0923 20:10:07.602870 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0923 20:10:07.603004 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:10:07.603493 (+ 489us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:10:07.603497 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:10:07.603880 (+ 383us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:10:07.604047 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:10:07.604055 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:10:07.605146 (+ 1091us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:10:07.605152 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:10:07.606645 (+ 1493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:10:07.606661 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:10:07.607508 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:10:07.607515 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:10:07.607694 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:10:07.614972 (+ 7278us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:10:07.615002 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:10:07.619314 (+ 4312us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.170.190:36595: Server oryx/127.1.170.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":399,"thread_start_us":288,"threads_started":1}
W20260923 20:10:07.619918 11002 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 20:10:07.602458 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.170.130:39129 (local address 127.1.170.190:36595)
0923 20:10:07.602690 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0923 20:10:07.602696 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:10:07.602965 (+ 269us) server_negotiation.cc:407] Connection header received
0923 20:10:07.603167 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:10:07.603171 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:10:07.603316 (+ 145us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:10:07.603405 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:10:07.604203 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:10:07.604984 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:10:07.606826 (+ 1842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:10:07.607310 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:10:07.607414 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:10:07.619556 (+ 12142us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.170.130:39129: BlockingRecv error: recv got EOF from 127.1.170.130:39129 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Sep 23 20:10:08 dist-test-slave-th4g krb5kdc[9971](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.170.129@KRBTEST.COM for oryx/127.1.170.190@KRBTEST.COM, Server not found in Kerberos database
Sep 23 20:10:08 dist-test-slave-th4g krb5kdc[9971](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.170.129@KRBTEST.COM for oryx/127.1.170.190@KRBTEST.COM, Server not found in Kerberos database
I20260923 20:10:08.132086 11005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:10:08.119736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.190:36595 (local address 127.1.170.129:38621)
0923 20:10:08.120437 (+ 701us) negotiation.cc:107] Waiting for socket to connect
0923 20:10:08.120455 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0923 20:10:08.120587 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:10:08.120944 (+ 357us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:10:08.120947 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:10:08.121430 (+ 483us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:10:08.121617 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:10:08.121626 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:10:08.123259 (+ 1633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:10:08.123263 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:10:08.124961 (+ 1698us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:10:08.124976 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:10:08.125859 (+ 883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:10:08.125864 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:10:08.126025 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:10:08.126851 (+ 826us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:10:08.126877 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:10:08.131817 (+ 4940us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.170.190:36595: Server oryx/127.1.170.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":559,"thread_start_us":319,"threads_started":1}
W20260923 20:10:08.132513 11002 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 20:10:08.119875 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.170.129:38621 (local address 127.1.170.190:36595)
0923 20:10:08.120109 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0923 20:10:08.120118 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 20:10:08.120564 (+ 446us) server_negotiation.cc:407] Connection header received
0923 20:10:08.120716 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:10:08.120721 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:10:08.120795 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:10:08.120891 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:10:08.121830 (+ 939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:10:08.123023 (+ 1193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:10:08.125127 (+ 2104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:10:08.125655 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:10:08.125770 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:10:08.132179 (+ 6409us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.170.129:38621: BlockingRecv error: recv got EOF from 127.1.170.129:38621 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
Sep 23 20:10:08 dist-test-slave-th4g krb5kdc[9971](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.170.131@KRBTEST.COM for oryx/127.1.170.190@KRBTEST.COM, Server not found in Kerberos database
Sep 23 20:10:08 dist-test-slave-th4g krb5kdc[9971](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.170.131@KRBTEST.COM for oryx/127.1.170.190@KRBTEST.COM, Server not found in Kerberos database
I20260923 20:10:08.150652 11006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:10:08.140062 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.190:36595 (local address 127.1.170.131:37093)
0923 20:10:08.140653 (+ 591us) negotiation.cc:107] Waiting for socket to connect
0923 20:10:08.140670 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0923 20:10:08.140867 (+ 197us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:10:08.141218 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:10:08.141222 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:10:08.141594 (+ 372us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:10:08.141805 (+ 211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:10:08.141814 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:10:08.143063 (+ 1249us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:10:08.143067 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:10:08.144117 (+ 1050us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:10:08.144130 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:10:08.144902 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:10:08.144908 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:10:08.145045 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:10:08.145927 (+ 882us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:10:08.145960 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:10:08.150298 (+ 4338us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.170.190:36595: Server oryx/127.1.170.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":411,"thread_start_us":314,"threads_started":1}
W20260923 20:10:08.150826 11002 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 20:10:08.140156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.170.131:37093 (local address 127.1.170.190:36595)
0923 20:10:08.140418 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0923 20:10:08.140425 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:10:08.140838 (+ 413us) server_negotiation.cc:407] Connection header received
0923 20:10:08.140967 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:10:08.140971 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:10:08.141042 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:10:08.141143 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:10:08.142095 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:10:08.142916 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:10:08.144285 (+ 1369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:10:08.144731 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:10:08.144836 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:10:08.150532 (+ 5696us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.170.131:37093: BlockingRecv error: recv got EOF from 127.1.170.131:37093 (error 108)
Metrics: {"server-negotiator.queue_time_us":104}
Sep 23 20:10:08 dist-test-slave-th4g krb5kdc[9971](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.170.130@KRBTEST.COM for oryx/127.1.170.190@KRBTEST.COM, Server not found in Kerberos database
Sep 23 20:10:08 dist-test-slave-th4g krb5kdc[9971](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.170.130@KRBTEST.COM for oryx/127.1.170.190@KRBTEST.COM, Server not found in Kerberos database
I20260923 20:10:08.629850 11007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:10:08.620755 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.190:36595 (local address 127.1.170.130:57965)
0923 20:10:08.621349 (+ 594us) negotiation.cc:107] Waiting for socket to connect
0923 20:10:08.621365 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0923 20:10:08.621477 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:10:08.621812 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:10:08.621817 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:10:08.622302 (+ 485us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:10:08.622462 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:10:08.622470 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:10:08.623552 (+ 1082us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:10:08.623556 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:10:08.624473 (+ 917us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:10:08.624484 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:10:08.625229 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:10:08.625233 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:10:08.625336 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:10:08.625994 (+ 658us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:10:08.626012 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:10:08.629481 (+ 3469us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.170.190:36595: Server oryx/127.1.170.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":489,"thread_start_us":310,"threads_started":1}
W20260923 20:10:08.630002 11002 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 20:10:08.620880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.170.130:57965 (local address 127.1.170.190:36595)
0923 20:10:08.621109 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0923 20:10:08.621114 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 20:10:08.621447 (+ 333us) server_negotiation.cc:407] Connection header received
0923 20:10:08.621579 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:10:08.621583 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:10:08.621657 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:10:08.621752 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:10:08.622602 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:10:08.623397 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:10:08.624617 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:10:08.625063 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:10:08.625164 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:10:08.629665 (+ 4501us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.170.130:57965: BlockingRecv error: recv got EOF from 127.1.170.130:57965 (error 108)
Metrics: {"server-negotiator.queue_time_us":112}
Sep 23 20:10:09 dist-test-slave-th4g krb5kdc[9971](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.170.129@KRBTEST.COM for oryx/127.1.170.190@KRBTEST.COM, Server not found in Kerberos database
Sep 23 20:10:09 dist-test-slave-th4g krb5kdc[9971](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.170.129@KRBTEST.COM for oryx/127.1.170.190@KRBTEST.COM, Server not found in Kerberos database
I20260923 20:10:09.143196 11008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:10:09.133225 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.190:36595 (local address 127.1.170.129:38911)
0923 20:10:09.133739 (+ 514us) negotiation.cc:107] Waiting for socket to connect
0923 20:10:09.133756 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0923 20:10:09.133864 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:10:09.134278 (+ 414us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:10:09.134281 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:10:09.134658 (+ 377us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:10:09.134888 (+ 230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:10:09.134922 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:10:09.136024 (+ 1102us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:10:09.136029 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:10:09.137222 (+ 1193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:10:09.137236 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:10:09.138098 (+ 862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:10:09.138103 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:10:09.138262 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:10:09.138955 (+ 693us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:10:09.138977 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:10:09.142837 (+ 3860us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.170.190:36595: Server oryx/127.1.170.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":371,"thread_start_us":274,"threads_started":1}
W20260923 20:10:09.143384 11009 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 20:10:09.133440 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.170.129:38911 (local address 127.1.170.190:36595)
0923 20:10:09.134018 (+ 578us) server_negotiation.cc:206] Beginning negotiation
0923 20:10:09.134023 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 20:10:09.134038 (+ 15us) server_negotiation.cc:407] Connection header received
0923 20:10:09.134082 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:10:09.134085 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:10:09.134139 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:10:09.134226 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:10:09.135062 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:10:09.135852 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:10:09.137374 (+ 1522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:10:09.137817 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:10:09.138025 (+ 208us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:10:09.143093 (+ 5068us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.170.129:38911: BlockingRecv error: recv got EOF from 127.1.170.129:38911 (error 108)
Metrics: {"server-negotiator.queue_time_us":493,"thread_start_us":368,"threads_started":1}
Sep 23 20:10:09 dist-test-slave-th4g krb5kdc[9971](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.170.131@KRBTEST.COM for oryx/127.1.170.190@KRBTEST.COM, Server not found in Kerberos database
Sep 23 20:10:09 dist-test-slave-th4g krb5kdc[9971](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.170.131@KRBTEST.COM for oryx/127.1.170.190@KRBTEST.COM, Server not found in Kerberos database
I20260923 20:10:09.161584 11010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:10:09.151859 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.190:36595 (local address 127.1.170.131:60805)
0923 20:10:09.152450 (+ 591us) negotiation.cc:107] Waiting for socket to connect
0923 20:10:09.152467 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0923 20:10:09.152576 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:10:09.153320 (+ 744us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:10:09.153323 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:10:09.153852 (+ 529us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:10:09.154148 (+ 296us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:10:09.154162 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:10:09.155269 (+ 1107us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:10:09.155273 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:10:09.156230 (+ 957us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:10:09.156240 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:10:09.156959 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:10:09.156963 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:10:09.157143 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:10:09.157745 (+ 602us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:10:09.157762 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:10:09.161323 (+ 3561us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.170.190:36595: Server oryx/127.1.170.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":483,"thread_start_us":273,"threads_started":1}
W20260923 20:10:09.161716 11009 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 20:10:09.152053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.170.131:60805 (local address 127.1.170.190:36595)
0923 20:10:09.152449 (+ 396us) server_negotiation.cc:206] Beginning negotiation
0923 20:10:09.152458 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 20:10:09.152544 (+ 86us) server_negotiation.cc:407] Connection header received
0923 20:10:09.153023 (+ 479us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:10:09.153028 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:10:09.153121 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:10:09.153248 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:10:09.154349 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:10:09.155112 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:10:09.156369 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:10:09.156795 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:10:09.156895 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:10:09.161506 (+ 4611us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.170.131:60805: BlockingRecv error: recv got EOF from 127.1.170.131:60805 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Sep 23 20:10:09 dist-test-slave-th4g krb5kdc[9971](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.170.130@KRBTEST.COM for oryx/127.1.170.190@KRBTEST.COM, Server not found in Kerberos database
Sep 23 20:10:09 dist-test-slave-th4g krb5kdc[9971](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.170.130@KRBTEST.COM for oryx/127.1.170.190@KRBTEST.COM, Server not found in Kerberos database
I20260923 20:10:09.640322 11011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:10:09.630968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.190:36595 (local address 127.1.170.130:33763)
0923 20:10:09.631483 (+ 515us) negotiation.cc:107] Waiting for socket to connect
0923 20:10:09.631498 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0923 20:10:09.631653 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:10:09.631981 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:10:09.631986 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:10:09.632367 (+ 381us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:10:09.632518 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:10:09.632526 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:10:09.633714 (+ 1188us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:10:09.633719 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:10:09.634706 (+ 987us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:10:09.634717 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:10:09.635448 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:10:09.635454 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:10:09.635611 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:10:09.636404 (+ 793us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:10:09.636426 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:10:09.640056 (+ 3630us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.170.190:36595: Server oryx/127.1.170.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":408,"thread_start_us":268,"threads_started":1}
W20260923 20:10:09.640571 11009 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 20:10:09.631033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.170.130:33763 (local address 127.1.170.190:36595)
0923 20:10:09.631256 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0923 20:10:09.631265 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 20:10:09.631582 (+ 317us) server_negotiation.cc:407] Connection header received
0923 20:10:09.631789 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:10:09.631793 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:10:09.631853 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:10:09.631958 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:10:09.632651 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:10:09.633568 (+ 917us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:10:09.634861 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:10:09.635258 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:10:09.635410 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:10:09.640247 (+ 4837us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.170.130:33763: BlockingRecv error: recv got EOF from 127.1.170.130:33763 (error 108)
Metrics: {"server-negotiator.queue_time_us":83}
Sep 23 20:10:10 dist-test-slave-th4g krb5kdc[9971](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.170.129@KRBTEST.COM for oryx/127.1.170.190@KRBTEST.COM, Server not found in Kerberos database
Sep 23 20:10:10 dist-test-slave-th4g krb5kdc[9971](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.170.129@KRBTEST.COM for oryx/127.1.170.190@KRBTEST.COM, Server not found in Kerberos database
I20260923 20:10:10.154302 11012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:10:10.144221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.190:36595 (local address 127.1.170.129:37337)
0923 20:10:10.144879 (+ 658us) negotiation.cc:107] Waiting for socket to connect
0923 20:10:10.144898 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0923 20:10:10.145018 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:10:10.145416 (+ 398us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:10:10.145420 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:10:10.145836 (+ 416us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:10:10.146058 (+ 222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:10:10.146066 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:10:10.147105 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:10:10.147109 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:10:10.148345 (+ 1236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:10:10.148357 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:10:10.149249 (+ 892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:10:10.149255 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:10:10.149452 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:10:10.150231 (+ 779us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:10:10.150251 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:10:10.154037 (+ 3786us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.170.190:36595: Server oryx/127.1.170.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":359,"thread_start_us":234,"threads_started":1}
W20260923 20:10:10.154461 11013 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 20:10:10.144270 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.170.129:37337 (local address 127.1.170.190:36595)
0923 20:10:10.145005 (+ 735us) server_negotiation.cc:206] Beginning negotiation
0923 20:10:10.145009 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 20:10:10.145027 (+ 18us) server_negotiation.cc:407] Connection header received
0923 20:10:10.145177 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:10:10.145181 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:10:10.145253 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:10:10.145351 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:10:10.146211 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:10:10.146941 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:10:10.148483 (+ 1542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:10:10.149069 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:10:10.149222 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:10:10.154242 (+ 5020us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.170.129:37337: BlockingRecv error: recv got EOF from 127.1.170.129:37337 (error 108)
Metrics: {"server-negotiator.queue_time_us":643,"thread_start_us":541,"threads_started":1}
Sep 23 20:10:10 dist-test-slave-th4g krb5kdc[9971](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.170.131@KRBTEST.COM for oryx/127.1.170.190@KRBTEST.COM, Server not found in Kerberos database
Sep 23 20:10:10 dist-test-slave-th4g krb5kdc[9971](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.170.131@KRBTEST.COM for oryx/127.1.170.190@KRBTEST.COM, Server not found in Kerberos database
I20260923 20:10:10.171846 11014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:10:10.162630 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.190:36595 (local address 127.1.170.131:45875)
0923 20:10:10.163175 (+ 545us) negotiation.cc:107] Waiting for socket to connect
0923 20:10:10.163190 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0923 20:10:10.163313 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:10:10.163584 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:10:10.163587 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:10:10.163966 (+ 379us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:10:10.164115 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:10:10.164123 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:10:10.165183 (+ 1060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:10:10.165187 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:10:10.166273 (+ 1086us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:10:10.166284 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:10:10.167217 (+ 933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:10:10.167222 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:10:10.167342 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:10:10.167973 (+ 631us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:10:10.167990 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:10:10.171498 (+ 3508us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.170.190:36595: Server oryx/127.1.170.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":415,"thread_start_us":291,"threads_started":1}
W20260923 20:10:10.172098 11013 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 20:10:10.162698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.170.131:45875 (local address 127.1.170.190:36595)
0923 20:10:10.162848 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0923 20:10:10.162853 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 20:10:10.163279 (+ 426us) server_negotiation.cc:407] Connection header received
0923 20:10:10.163406 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:10:10.163410 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:10:10.163470 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:10:10.163552 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:10:10.164274 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:10:10.165016 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:10:10.166410 (+ 1394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:10:10.167052 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:10:10.167172 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:10:10.171796 (+ 4624us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.170.131:45875: BlockingRecv error: recv got EOF from 127.1.170.131:45875 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Sep 23 20:10:10 dist-test-slave-th4g krb5kdc[9971](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.170.130@KRBTEST.COM for oryx/127.1.170.190@KRBTEST.COM, Server not found in Kerberos database
Sep 23 20:10:10 dist-test-slave-th4g krb5kdc[9971](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.170.130@KRBTEST.COM for oryx/127.1.170.190@KRBTEST.COM, Server not found in Kerberos database
I20260923 20:10:10.652259 11015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:10:10.641358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.190:36595 (local address 127.1.170.130:38203)
0923 20:10:10.642234 (+ 876us) negotiation.cc:107] Waiting for socket to connect
0923 20:10:10.642249 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0923 20:10:10.642464 (+ 215us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:10:10.643000 (+ 536us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:10:10.643004 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:10:10.643545 (+ 541us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:10:10.643697 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:10:10.643706 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:10:10.645014 (+ 1308us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:10:10.645022 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:10:10.645982 (+ 960us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:10:10.645993 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:10:10.646738 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:10:10.646743 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:10:10.646916 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:10:10.647604 (+ 688us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:10:10.647624 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:10:10.651985 (+ 4361us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.170.190:36595: Server oryx/127.1.170.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":791,"thread_start_us":398,"threads_started":1}
W20260923 20:10:10.652602 11013 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 20:10:10.641599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.170.130:38203 (local address 127.1.170.190:36595)
0923 20:10:10.642008 (+ 409us) server_negotiation.cc:206] Beginning negotiation
0923 20:10:10.642016 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 20:10:10.642414 (+ 398us) server_negotiation.cc:407] Connection header received
0923 20:10:10.642609 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:10:10.642615 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:10:10.642744 (+ 129us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:10:10.642873 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:10:10.643850 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:10:10.644882 (+ 1032us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:10:10.646139 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:10:10.646583 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:10:10.646725 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:10:10.652270 (+ 5545us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.170.130:38203: BlockingRecv error: recv got EOF from 127.1.170.130:38203 (error 108)
Metrics: {"server-negotiator.queue_time_us":116}
Sep 23 20:10:11 dist-test-slave-th4g krb5kdc[9971](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.170.129@KRBTEST.COM for oryx/127.1.170.190@KRBTEST.COM, Server not found in Kerberos database
Sep 23 20:10:11 dist-test-slave-th4g krb5kdc[9971](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.170.129@KRBTEST.COM for oryx/127.1.170.190@KRBTEST.COM, Server not found in Kerberos database
I20260923 20:10:11.165196 11016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:10:11.155342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.190:36595 (local address 127.1.170.129:60175)
0923 20:10:11.155818 (+ 476us) negotiation.cc:107] Waiting for socket to connect
0923 20:10:11.155837 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0923 20:10:11.155947 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:10:11.156423 (+ 476us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:10:11.156427 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:10:11.156773 (+ 346us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:10:11.156931 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:10:11.156939 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:10:11.158011 (+ 1072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:10:11.158016 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:10:11.159024 (+ 1008us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:10:11.159036 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:10:11.159777 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:10:11.159782 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:10:11.159904 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:10:11.160696 (+ 792us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:10:11.160723 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:10:11.164909 (+ 4186us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.170.190:36595: Server oryx/127.1.170.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":362,"thread_start_us":272,"threads_started":1}
W20260923 20:10:11.165586 11017 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 20:10:11.155451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.170.129:60175 (local address 127.1.170.190:36595)
0923 20:10:11.156144 (+ 693us) server_negotiation.cc:206] Beginning negotiation
0923 20:10:11.156149 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 20:10:11.156164 (+ 15us) server_negotiation.cc:407] Connection header received
0923 20:10:11.156212 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:10:11.156215 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:10:11.156272 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:10:11.156360 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:10:11.157092 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:10:11.157809 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:10:11.159166 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:10:11.159610 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:10:11.159709 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:10:11.165067 (+ 5358us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.170.129:60175: BlockingRecv error: recv got EOF from 127.1.170.129:60175 (error 108)
Metrics: {"server-negotiator.queue_time_us":597,"thread_start_us":515,"threads_started":1}
Sep 23 20:10:11 dist-test-slave-th4g krb5kdc[9971](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.170.131@KRBTEST.COM for oryx/127.1.170.190@KRBTEST.COM, Server not found in Kerberos database
Sep 23 20:10:11 dist-test-slave-th4g krb5kdc[9971](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.170.131@KRBTEST.COM for oryx/127.1.170.190@KRBTEST.COM, Server not found in Kerberos database
I20260923 20:10:11.183241 11018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:10:11.173023 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.190:36595 (local address 127.1.170.131:50137)
0923 20:10:11.173636 (+ 613us) negotiation.cc:107] Waiting for socket to connect
0923 20:10:11.173659 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0923 20:10:11.173814 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:10:11.174300 (+ 486us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:10:11.174305 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:10:11.174854 (+ 549us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:10:11.175012 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:10:11.175021 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:10:11.176118 (+ 1097us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:10:11.176123 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:10:11.177217 (+ 1094us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:10:11.177229 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:10:11.178473 (+ 1244us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:10:11.178478 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:10:11.178611 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:10:11.179235 (+ 624us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:10:11.179253 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:10:11.183002 (+ 3749us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.170.190:36595: Server oryx/127.1.170.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":484,"thread_start_us":251,"threads_started":1}
W20260923 20:10:11.183956 11017 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 20:10:11.173166 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.170.131:50137 (local address 127.1.170.190:36595)
0923 20:10:11.173436 (+ 270us) server_negotiation.cc:206] Beginning negotiation
0923 20:10:11.173444 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 20:10:11.173777 (+ 333us) server_negotiation.cc:407] Connection header received
0923 20:10:11.173996 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:10:11.174001 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:10:11.174083 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:10:11.174246 (+ 163us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:10:11.175203 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:10:11.175969 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:10:11.177369 (+ 1400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:10:11.178231 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:10:11.178407 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:10:11.183715 (+ 5308us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.170.131:50137: BlockingRecv error: recv got EOF from 127.1.170.131:50137 (error 108)
Metrics: {"server-negotiator.queue_time_us":143}
Sep 23 20:10:11 dist-test-slave-th4g krb5kdc[9971](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.170.130@KRBTEST.COM for oryx/127.1.170.190@KRBTEST.COM, Server not found in Kerberos database
Sep 23 20:10:11 dist-test-slave-th4g krb5kdc[9971](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.170.130@KRBTEST.COM for oryx/127.1.170.190@KRBTEST.COM, Server not found in Kerberos database
I20260923 20:10:11.665079 11019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:10:11.653691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.190:36595 (local address 127.1.170.130:38501)
0923 20:10:11.654671 (+ 980us) negotiation.cc:107] Waiting for socket to connect
0923 20:10:11.654692 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0923 20:10:11.654864 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:10:11.655438 (+ 574us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:10:11.655444 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:10:11.655845 (+ 401us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:10:11.656028 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:10:11.656038 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:10:11.657450 (+ 1412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:10:11.657454 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:10:11.658783 (+ 1329us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:10:11.658797 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:10:11.659997 (+ 1200us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:10:11.660003 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:10:11.660129 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:10:11.660836 (+ 707us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:10:11.660855 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:10:11.664807 (+ 3952us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.170.190:36595: Server oryx/127.1.170.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":691,"thread_start_us":548,"threads_started":1}
W20260923 20:10:11.665354 11017 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 20:10:11.653803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.170.130:38501 (local address 127.1.170.190:36595)
0923 20:10:11.654080 (+ 277us) server_negotiation.cc:206] Beginning negotiation
0923 20:10:11.654087 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:10:11.654925 (+ 838us) server_negotiation.cc:407] Connection header received
0923 20:10:11.655031 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:10:11.655035 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:10:11.655143 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:10:11.655318 (+ 175us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:10:11.656219 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:10:11.657289 (+ 1070us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:10:11.659022 (+ 1733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:10:11.659759 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:10:11.659930 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:10:11.665086 (+ 5156us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.170.130:38501: BlockingRecv error: recv got EOF from 127.1.170.130:38501 (error 108)
Metrics: {"server-negotiator.queue_time_us":108}
Sep 23 20:10:12 dist-test-slave-th4g krb5kdc[9971](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.170.129@KRBTEST.COM for oryx/127.1.170.190@KRBTEST.COM, Server not found in Kerberos database
Sep 23 20:10:12 dist-test-slave-th4g krb5kdc[9971](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.170.129@KRBTEST.COM for oryx/127.1.170.190@KRBTEST.COM, Server not found in Kerberos database
I20260923 20:10:12.178086 11020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:10:12.166461 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.190:36595 (local address 127.1.170.129:33675)
0923 20:10:12.167433 (+ 972us) negotiation.cc:107] Waiting for socket to connect
0923 20:10:12.167449 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0923 20:10:12.167565 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:10:12.167971 (+ 406us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:10:12.167975 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:10:12.168748 (+ 773us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:10:12.168923 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:10:12.168932 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:10:12.170484 (+ 1552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:10:12.170489 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:10:12.171657 (+ 1168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:10:12.171670 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:10:12.172831 (+ 1161us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:10:12.172835 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:10:12.172956 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:10:12.173723 (+ 767us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:10:12.173742 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:10:12.177778 (+ 4036us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.170.190:36595: Server oryx/127.1.170.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":846,"thread_start_us":761,"threads_started":1}
W20260923 20:10:12.178515 11021 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 20:10:12.166742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.170.129:33675 (local address 127.1.170.190:36595)
0923 20:10:12.167250 (+ 508us) server_negotiation.cc:206] Beginning negotiation
0923 20:10:12.167257 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:10:12.167534 (+ 277us) server_negotiation.cc:407] Connection header received
0923 20:10:12.167675 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:10:12.167681 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:10:12.167780 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:10:12.167911 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:10:12.169132 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:10:12.170286 (+ 1154us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:10:12.171891 (+ 1605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:10:12.172658 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:10:12.172773 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:10:12.178231 (+ 5458us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.170.129:33675: BlockingRecv error: recv got EOF from 127.1.170.129:33675 (error 108)
Metrics: {"server-negotiator.queue_time_us":373,"thread_start_us":256,"threads_started":1}
Sep 23 20:10:12 dist-test-slave-th4g krb5kdc[9971](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.170.131@KRBTEST.COM for oryx/127.1.170.190@KRBTEST.COM, Server not found in Kerberos database
Sep 23 20:10:12 dist-test-slave-th4g krb5kdc[9971](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.170.131@KRBTEST.COM for oryx/127.1.170.190@KRBTEST.COM, Server not found in Kerberos database
I20260923 20:10:12.193629 11022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:10:12.184547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.190:36595 (local address 127.1.170.131:59445)
0923 20:10:12.185083 (+ 536us) negotiation.cc:107] Waiting for socket to connect
0923 20:10:12.185100 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0923 20:10:12.185209 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:10:12.185520 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:10:12.185525 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:10:12.186039 (+ 514us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:10:12.186202 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:10:12.186211 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:10:12.187213 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:10:12.187220 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:10:12.188190 (+ 970us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:10:12.188201 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:10:12.188957 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:10:12.188964 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:10:12.189126 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:10:12.189791 (+ 665us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:10:12.189807 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:10:12.193287 (+ 3480us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.170.190:36595: Server oryx/127.1.170.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":426,"thread_start_us":316,"threads_started":1}
W20260923 20:10:12.193717 11021 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 20:10:12.184668 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.170.131:59445 (local address 127.1.170.190:36595)
0923 20:10:12.184893 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0923 20:10:12.184898 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 20:10:12.185179 (+ 281us) server_negotiation.cc:407] Connection header received
0923 20:10:12.185310 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:10:12.185314 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:10:12.185375 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:10:12.185490 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:10:12.186343 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:10:12.187071 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:10:12.188374 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:10:12.188782 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:10:12.188922 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:10:12.193479 (+ 4557us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.170.131:59445: BlockingRecv error: recv got EOF from 127.1.170.131:59445 (error 108)
Metrics: {"server-negotiator.queue_time_us":117}
Sep 23 20:10:12 dist-test-slave-th4g krb5kdc[9971](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.170.130@KRBTEST.COM for oryx/127.1.170.190@KRBTEST.COM, Server not found in Kerberos database
Sep 23 20:10:12 dist-test-slave-th4g krb5kdc[9971](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.170.130@KRBTEST.COM for oryx/127.1.170.190@KRBTEST.COM, Server not found in Kerberos database
I20260923 20:10:12.679840 11023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:10:12.666060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.190:36595 (local address 127.1.170.130:57937)
0923 20:10:12.666566 (+ 506us) negotiation.cc:107] Waiting for socket to connect
0923 20:10:12.666581 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0923 20:10:12.666748 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:10:12.667106 (+ 358us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:10:12.667111 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:10:12.667510 (+ 399us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:10:12.667681 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:10:12.667689 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:10:12.668709 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:10:12.668717 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:10:12.669615 (+ 898us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:10:12.669626 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:10:12.670388 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:10:12.670393 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:10:12.670569 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:10:12.675830 (+ 5261us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:10:12.675850 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:10:12.679513 (+ 3663us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.170.190:36595: Server oryx/127.1.170.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":381,"thread_start_us":266,"threads_started":1}
W20260923 20:10:12.680022 11021 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 20:10:12.666288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.170.130:57937 (local address 127.1.170.190:36595)
0923 20:10:12.666491 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0923 20:10:12.666498 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:10:12.666668 (+ 170us) server_negotiation.cc:407] Connection header received
0923 20:10:12.666877 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:10:12.666882 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:10:12.666968 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:10:12.667077 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:10:12.667832 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:10:12.668583 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:10:12.669754 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:10:12.670239 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:10:12.670376 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:10:12.679746 (+ 9370us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.170.130:57937: BlockingRecv error: recv got EOF from 127.1.170.130:57937 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
Sep 23 20:10:13 dist-test-slave-th4g krb5kdc[9971](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.170.129@KRBTEST.COM for oryx/127.1.170.190@KRBTEST.COM, Server not found in Kerberos database
Sep 23 20:10:13 dist-test-slave-th4g krb5kdc[9971](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.170.129@KRBTEST.COM for oryx/127.1.170.190@KRBTEST.COM, Server not found in Kerberos database
I20260923 20:10:13.188586 11024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:10:13.179119 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.190:36595 (local address 127.1.170.129:33011)
0923 20:10:13.179742 (+ 623us) negotiation.cc:107] Waiting for socket to connect
0923 20:10:13.179758 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0923 20:10:13.179917 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:10:13.180259 (+ 342us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:10:13.180264 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:10:13.180694 (+ 430us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:10:13.180857 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:10:13.180865 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:10:13.181867 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:10:13.181903 (+ 36us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:10:13.183327 (+ 1424us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:10:13.183340 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:10:13.184062 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:10:13.184067 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:10:13.184227 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:10:13.184875 (+ 648us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:10:13.184892 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:10:13.188209 (+ 3317us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.170.190:36595: Server oryx/127.1.170.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":525,"thread_start_us":388,"threads_started":1}
W20260923 20:10:13.188642 11021 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 20:10:13.179208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.170.129:33011 (local address 127.1.170.190:36595)
0923 20:10:13.179564 (+ 356us) server_negotiation.cc:206] Beginning negotiation
0923 20:10:13.179572 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 20:10:13.179843 (+ 271us) server_negotiation.cc:407] Connection header received
0923 20:10:13.180043 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:10:13.180053 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:10:13.180127 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:10:13.180235 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:10:13.180996 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:10:13.181721 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:10:13.183472 (+ 1751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:10:13.183894 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:10:13.183995 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:10:13.188432 (+ 4437us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.170.129:33011: BlockingRecv error: recv got EOF from 127.1.170.129:33011 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
Sep 23 20:10:13 dist-test-slave-th4g krb5kdc[9971](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.170.131@KRBTEST.COM for oryx/127.1.170.190@KRBTEST.COM, Server not found in Kerberos database
Sep 23 20:10:13 dist-test-slave-th4g krb5kdc[9971](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.170.131@KRBTEST.COM for oryx/127.1.170.190@KRBTEST.COM, Server not found in Kerberos database
I20260923 20:10:13.202883 11025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:10:13.194798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.190:36595 (local address 127.1.170.131:35791)
0923 20:10:13.195300 (+ 502us) negotiation.cc:107] Waiting for socket to connect
0923 20:10:13.195322 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0923 20:10:13.195472 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:10:13.195760 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:10:13.195763 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:10:13.196108 (+ 345us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:10:13.196280 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:10:13.196288 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:10:13.197180 (+ 892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:10:13.197184 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:10:13.198171 (+ 987us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:10:13.198182 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:10:13.198810 (+ 628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:10:13.198814 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:10:13.198924 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:10:13.199482 (+ 558us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:10:13.199498 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:10:13.202659 (+ 3161us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.170.190:36595: Server oryx/127.1.170.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":361,"thread_start_us":258,"threads_started":1}
W20260923 20:10:13.203024 11021 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 20:10:13.194832 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.170.131:35791 (local address 127.1.170.190:36595)
0923 20:10:13.195003 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0923 20:10:13.195012 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 20:10:13.195405 (+ 393us) server_negotiation.cc:407] Connection header received
0923 20:10:13.195575 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:10:13.195578 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:10:13.195643 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:10:13.195715 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:10:13.196391 (+ 676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:10:13.197066 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:10:13.198287 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:10:13.198688 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:10:13.198778 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:10:13.202822 (+ 4044us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.170.131:35791: BlockingRecv error: recv got EOF from 127.1.170.131:35791 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Sep 23 20:10:13 dist-test-slave-th4g krb5kdc[9971](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.170.130@KRBTEST.COM for oryx/127.1.170.190@KRBTEST.COM, Server not found in Kerberos database
Sep 23 20:10:13 dist-test-slave-th4g krb5kdc[9971](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.170.130@KRBTEST.COM for oryx/127.1.170.190@KRBTEST.COM, Server not found in Kerberos database
I20260923 20:10:13.689407 11026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:10:13.680697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.190:36595 (local address 127.1.170.130:57561)
0923 20:10:13.681224 (+ 527us) negotiation.cc:107] Waiting for socket to connect
0923 20:10:13.681240 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0923 20:10:13.681373 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:10:13.681647 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:10:13.681650 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:10:13.682042 (+ 392us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:10:13.682200 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:10:13.682208 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:10:13.683110 (+ 902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:10:13.683113 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:10:13.684010 (+ 897us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:10:13.684020 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:10:13.684717 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:10:13.684724 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:10:13.684869 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:10:13.685578 (+ 709us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:10:13.685595 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:10:13.689048 (+ 3453us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.170.190:36595: Server oryx/127.1.170.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":409,"thread_start_us":309,"threads_started":1}
W20260923 20:10:13.689446 11021 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 20:10:13.680801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.170.130:57561 (local address 127.1.170.190:36595)
0923 20:10:13.680969 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0923 20:10:13.680974 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 20:10:13.681326 (+ 352us) server_negotiation.cc:407] Connection header received
0923 20:10:13.681465 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:10:13.681468 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:10:13.681524 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:10:13.681601 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:10:13.682322 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:10:13.682997 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:10:13.684114 (+ 1117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:10:13.684568 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:10:13.684658 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:10:13.689228 (+ 4570us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.170.130:57561: BlockingRecv error: recv got EOF from 127.1.170.130:57561 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Sep 23 20:10:14 dist-test-slave-th4g krb5kdc[9971](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.170.129@KRBTEST.COM for oryx/127.1.170.190@KRBTEST.COM, Server not found in Kerberos database
Sep 23 20:10:14 dist-test-slave-th4g krb5kdc[9971](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.170.129@KRBTEST.COM for oryx/127.1.170.190@KRBTEST.COM, Server not found in Kerberos database
I20260923 20:10:14.198477 11027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:10:14.189555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.190:36595 (local address 127.1.170.129:40629)
0923 20:10:14.190106 (+ 551us) negotiation.cc:107] Waiting for socket to connect
0923 20:10:14.190122 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0923 20:10:14.190237 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:10:14.190537 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:10:14.190540 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:10:14.190912 (+ 372us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:10:14.191049 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:10:14.191056 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:10:14.192087 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:10:14.192091 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:10:14.193054 (+ 963us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:10:14.193065 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:10:14.193751 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:10:14.193754 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:10:14.193871 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:10:14.194576 (+ 705us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:10:14.194593 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:10:14.198250 (+ 3657us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.170.190:36595: Server oryx/127.1.170.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":442,"thread_start_us":344,"threads_started":1}
W20260923 20:10:14.198664 11021 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 20:10:14.189640 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.170.129:40629 (local address 127.1.170.190:36595)
0923 20:10:14.189832 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0923 20:10:14.189839 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:10:14.190220 (+ 381us) server_negotiation.cc:407] Connection header received
0923 20:10:14.190339 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:10:14.190343 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:10:14.190403 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:10:14.190487 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:10:14.191207 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:10:14.191950 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:10:14.193216 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:10:14.193618 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:10:14.193731 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:10:14.198451 (+ 4720us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.170.129:40629: BlockingRecv error: recv got EOF from 127.1.170.129:40629 (error 108)
Metrics: {"server-negotiator.queue_time_us":30}
Sep 23 20:10:14 dist-test-slave-th4g krb5kdc[9971](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.170.131@KRBTEST.COM for oryx/127.1.170.190@KRBTEST.COM, Server not found in Kerberos database
Sep 23 20:10:14 dist-test-slave-th4g krb5kdc[9971](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.170.131@KRBTEST.COM for oryx/127.1.170.190@KRBTEST.COM, Server not found in Kerberos database
I20260923 20:10:14.212791 11028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:10:14.203701 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.190:36595 (local address 127.1.170.131:54181)
0923 20:10:14.204209 (+ 508us) negotiation.cc:107] Waiting for socket to connect
0923 20:10:14.204224 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0923 20:10:14.204375 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:10:14.204704 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:10:14.204708 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:10:14.205104 (+ 396us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:10:14.205266 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:10:14.205274 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:10:14.206293 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:10:14.206299 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:10:14.207253 (+ 954us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:10:14.207264 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:10:14.208351 (+ 1087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:10:14.208361 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:10:14.208567 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:10:14.209255 (+ 688us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:10:14.209272 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:10:14.212560 (+ 3288us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.170.190:36595: Server oryx/127.1.170.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":422,"thread_start_us":269,"threads_started":1}
W20260923 20:10:14.212980 11021 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 20:10:14.203804 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.170.131:54181 (local address 127.1.170.190:36595)
0923 20:10:14.204021 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0923 20:10:14.204029 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 20:10:14.204357 (+ 328us) server_negotiation.cc:407] Connection header received
0923 20:10:14.204516 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:10:14.204525 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:10:14.204580 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:10:14.204680 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:10:14.205398 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:10:14.206155 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:10:14.207437 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:10:14.208124 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:10:14.208280 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:10:14.212759 (+ 4479us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.170.131:54181: BlockingRecv error: recv got EOF from 127.1.170.131:54181 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Sep 23 20:10:14 dist-test-slave-th4g krb5kdc[9971](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.170.130@KRBTEST.COM for oryx/127.1.170.190@KRBTEST.COM, Server not found in Kerberos database
Sep 23 20:10:14 dist-test-slave-th4g krb5kdc[9971](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.170.130@KRBTEST.COM for oryx/127.1.170.190@KRBTEST.COM, Server not found in Kerberos database
I20260923 20:10:14.699796 11029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:10:14.690500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.190:36595 (local address 127.1.170.130:58165)
0923 20:10:14.691039 (+ 539us) negotiation.cc:107] Waiting for socket to connect
0923 20:10:14.691054 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0923 20:10:14.691208 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:10:14.691574 (+ 366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:10:14.691579 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:10:14.691998 (+ 419us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:10:14.692194 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:10:14.692205 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:10:14.693482 (+ 1277us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:10:14.693486 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:10:14.694594 (+ 1108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:10:14.694605 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:10:14.695295 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:10:14.695300 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:10:14.695416 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:10:14.695988 (+ 572us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:10:14.696004 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:10:14.699475 (+ 3471us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.170.190:36595: Server oryx/127.1.170.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":443,"thread_start_us":339,"threads_started":1}
W20260923 20:10:14.699846 11021 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 20:10:14.690735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.170.130:58165 (local address 127.1.170.190:36595)
0923 20:10:14.690944 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0923 20:10:14.690950 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:10:14.691134 (+ 184us) server_negotiation.cc:407] Connection header received
0923 20:10:14.691372 (+ 238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:10:14.691375 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:10:14.691433 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:10:14.691549 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:10:14.692350 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:10:14.693319 (+ 969us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:10:14.694723 (+ 1404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:10:14.695173 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:10:14.695269 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:10:14.699663 (+ 4394us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.170.130:58165: BlockingRecv error: recv got EOF from 127.1.170.130:58165 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
Sep 23 20:10:15 dist-test-slave-th4g krb5kdc[9971](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.170.129@KRBTEST.COM for oryx/127.1.170.190@KRBTEST.COM, Server not found in Kerberos database
Sep 23 20:10:15 dist-test-slave-th4g krb5kdc[9971](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.170.129@KRBTEST.COM for oryx/127.1.170.190@KRBTEST.COM, Server not found in Kerberos database
I20260923 20:10:15.208900 11030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:10:15.199646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.190:36595 (local address 127.1.170.129:44417)
0923 20:10:15.200170 (+ 524us) negotiation.cc:107] Waiting for socket to connect
0923 20:10:15.200187 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0923 20:10:15.200344 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:10:15.200674 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:10:15.200679 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:10:15.201046 (+ 367us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:10:15.201218 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:10:15.201227 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:10:15.202401 (+ 1174us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:10:15.202407 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:10:15.203455 (+ 1048us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:10:15.203466 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:10:15.204172 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:10:15.204177 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:10:15.204322 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:10:15.205013 (+ 691us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:10:15.205031 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:10:15.208658 (+ 3627us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.170.190:36595: Server oryx/127.1.170.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":384,"thread_start_us":283,"threads_started":1}
W20260923 20:10:15.209141 11021 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 20:10:15.199827 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.170.129:44417 (local address 127.1.170.190:36595)
0923 20:10:15.200001 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0923 20:10:15.200006 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 20:10:15.200281 (+ 275us) server_negotiation.cc:407] Connection header received
0923 20:10:15.200476 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:10:15.200480 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:10:15.200539 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:10:15.200643 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:10:15.201362 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:10:15.202272 (+ 910us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:10:15.203608 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:10:15.204028 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:10:15.204127 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:10:15.208895 (+ 4768us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.170.129:44417: BlockingRecv error: recv got EOF from 127.1.170.129:44417 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Sep 23 20:10:15 dist-test-slave-th4g krb5kdc[9971](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.170.131@KRBTEST.COM for oryx/127.1.170.190@KRBTEST.COM, Server not found in Kerberos database
Sep 23 20:10:15 dist-test-slave-th4g krb5kdc[9971](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.170.131@KRBTEST.COM for oryx/127.1.170.190@KRBTEST.COM, Server not found in Kerberos database
I20260923 20:10:15.222770 11031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:10:15.213927 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.190:36595 (local address 127.1.170.131:37907)
0923 20:10:15.214410 (+ 483us) negotiation.cc:107] Waiting for socket to connect
0923 20:10:15.214437 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0923 20:10:15.214588 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:10:15.214934 (+ 346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:10:15.214937 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:10:15.215446 (+ 509us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:10:15.215657 (+ 211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:10:15.215669 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:10:15.216725 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:10:15.216729 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:10:15.217656 (+ 927us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:10:15.217666 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:10:15.218385 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:10:15.218389 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:10:15.218507 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:10:15.219058 (+ 551us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:10:15.219075 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:10:15.222525 (+ 3450us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.170.190:36595: Server oryx/127.1.170.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":374,"thread_start_us":266,"threads_started":1}
W20260923 20:10:15.222951 11021 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 20:10:15.214036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.170.131:37907 (local address 127.1.170.190:36595)
0923 20:10:15.214375 (+ 339us) server_negotiation.cc:206] Beginning negotiation
0923 20:10:15.214382 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:10:15.214514 (+ 132us) server_negotiation.cc:407] Connection header received
0923 20:10:15.214721 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:10:15.214730 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:10:15.214787 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:10:15.214897 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:10:15.215810 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:10:15.216609 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:10:15.217779 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:10:15.218252 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:10:15.218349 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:10:15.222718 (+ 4369us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.170.131:37907: BlockingRecv error: recv got EOF from 127.1.170.131:37907 (error 108)
Metrics: {"server-negotiator.queue_time_us":225}
Sep 23 20:10:15 dist-test-slave-th4g krb5kdc[9971](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.170.130@KRBTEST.COM for oryx/127.1.170.190@KRBTEST.COM, Server not found in Kerberos database
Sep 23 20:10:15 dist-test-slave-th4g krb5kdc[9971](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.170.130@KRBTEST.COM for oryx/127.1.170.190@KRBTEST.COM, Server not found in Kerberos database
I20260923 20:10:15.711601 11032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:10:15.701081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.190:36595 (local address 127.1.170.130:43183)
0923 20:10:15.701658 (+ 577us) negotiation.cc:107] Waiting for socket to connect
0923 20:10:15.701674 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0923 20:10:15.701787 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:10:15.702177 (+ 390us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:10:15.702181 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:10:15.702577 (+ 396us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:10:15.702755 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:10:15.702793 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:10:15.704137 (+ 1344us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:10:15.704143 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:10:15.705559 (+ 1416us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:10:15.705575 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:10:15.706345 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:10:15.706350 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:10:15.706517 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:10:15.707268 (+ 751us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:10:15.707288 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:10:15.711255 (+ 3967us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.170.190:36595: Server oryx/127.1.170.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":450,"thread_start_us":319,"threads_started":1}
W20260923 20:10:15.711705 11021 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 20:10:15.701327 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.170.130:43183 (local address 127.1.170.190:36595)
0923 20:10:15.701488 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0923 20:10:15.701493 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 20:10:15.701786 (+ 293us) server_negotiation.cc:407] Connection header received
0923 20:10:15.701952 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:10:15.701956 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:10:15.702028 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:10:15.702108 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:10:15.702937 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:10:15.703847 (+ 910us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:10:15.705713 (+ 1866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:10:15.706169 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:10:15.706273 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:10:15.711470 (+ 5197us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.170.130:43183: BlockingRecv error: recv got EOF from 127.1.170.130:43183 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
I20260923 20:10:15.874135 1706 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskqUuku6/build/asan/bin/kudu pid 10560
I20260923 20:10:15.889005 11034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:10:15.875723 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34068 (local address 127.1.170.129:44063)
0923 20:10:15.876473 (+ 750us) server_negotiation.cc:206] Beginning negotiation
0923 20:10:15.876480 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:10:15.876508 (+ 28us) server_negotiation.cc:407] Connection header received
0923 20:10:15.876579 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:10:15.876586 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:10:15.876830 (+ 244us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:10:15.877014 (+ 184us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:10:15.878953 (+ 1939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:10:15.880035 (+ 1082us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:10:15.881367 (+ 1332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:10:15.882085 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:10:15.882211 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:10:15.885207 (+ 2996us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:10:15.885218 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:10:15.885235 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:10:15.885320 (+ 85us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:10:15.888427 (+ 3107us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.1.170.129@KRBTEST.COM not found in keytab (ticket kvno 2)
0923 20:10:15.888794 (+ 367us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:34068: Request ticket server kudu/127.1.170.129@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":577,"thread_start_us":413,"threads_started":1}
W20260923 20:10:15.889268 1706 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskqUuku6/build/asan/bin/kudu pid 10560: Not authorized: Client connection negotiation failed: client connection to 127.1.170.129:44063: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.1.170.129@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260923 20:10:15.889505 1706 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqUuku6/build/asan/bin/kudu with pid 10560
I20260923 20:10:15.906409 1706 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskqUuku6/build/asan/bin/kudu pid 10712
I20260923 20:10:15.921554 11035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:10:15.907480 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58608 (local address 127.1.170.130:45381)
0923 20:10:15.908529 (+ 1049us) server_negotiation.cc:206] Beginning negotiation
0923 20:10:15.908558 (+ 29us) server_negotiation.cc:399] Waiting for connection header
0923 20:10:15.908600 (+ 42us) server_negotiation.cc:407] Connection header received
0923 20:10:15.908680 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:10:15.908687 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:10:15.908934 (+ 247us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:10:15.909133 (+ 199us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:10:15.911101 (+ 1968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:10:15.912352 (+ 1251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:10:15.913794 (+ 1442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:10:15.914564 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:10:15.914687 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:10:15.917540 (+ 2853us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:10:15.917551 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:10:15.917568 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:10:15.917622 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:10:15.920953 (+ 3331us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.1.170.130@KRBTEST.COM not found in keytab (ticket kvno 2)
0923 20:10:15.921300 (+ 347us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:58608: Request ticket server kudu/127.1.170.130@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":712,"thread_start_us":526,"threads_started":1}
W20260923 20:10:15.921703 11035 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:58608: Request ticket server kudu/127.1.170.130@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260923 20:10:15.922098 1706 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskqUuku6/build/asan/bin/kudu pid 10712: Not authorized: Client connection negotiation failed: client connection to 127.1.170.130:45381: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.1.170.130@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260923 20:10:15.922322 1706 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqUuku6/build/asan/bin/kudu with pid 10712
I20260923 20:10:15.940050 1706 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskqUuku6/build/asan/bin/kudu pid 10845
I20260923 20:10:15.953614 11036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:10:15.941640 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55980 (local address 127.1.170.131:46717)
0923 20:10:15.942278 (+ 638us) server_negotiation.cc:206] Beginning negotiation
0923 20:10:15.942285 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:10:15.942311 (+ 26us) server_negotiation.cc:407] Connection header received
0923 20:10:15.942383 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:10:15.942390 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:10:15.942621 (+ 231us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:10:15.942846 (+ 225us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:10:15.944220 (+ 1374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:10:15.945286 (+ 1066us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:10:15.946658 (+ 1372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:10:15.947228 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:10:15.947398 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:10:15.950114 (+ 2716us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:10:15.950123 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:10:15.950138 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:10:15.950191 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:10:15.952875 (+ 2684us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.1.170.131@KRBTEST.COM not found in keytab (ticket kvno 2)
0923 20:10:15.953268 (+ 393us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:55980: Request ticket server kudu/127.1.170.131@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":466,"thread_start_us":350,"threads_started":1}
W20260923 20:10:15.953745 11036 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:55980: Request ticket server kudu/127.1.170.131@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260923 20:10:15.953760 1706 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskqUuku6/build/asan/bin/kudu pid 10845: Not authorized: Client connection negotiation failed: client connection to 127.1.170.131:46717: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.1.170.131@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260923 20:10:15.953963 1706 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqUuku6/build/asan/bin/kudu with pid 10845
I20260923 20:10:15.971592 1706 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskqUuku6/build/asan/bin/kudu pid 10491
I20260923 20:10:15.973212 10553 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:10:16.089609 1706 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqUuku6/build/asan/bin/kudu with pid 10491
2026-09-23T20:10:16Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (21633 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-09-23T20:10:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T20:10:16Z Disabled control of system clock
I20260923 20:10:16.169569 1706 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqUuku6/build/asan/bin/kudu
/tmp/dist-test-taskqUuku6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790194121577027-1706-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790194121577027-1706-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790194121577027-1706-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790194121577027-1706-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.170.190:39413
--webserver_interface=127.1.170.190
--webserver_port=0
--builtin_ntp_servers=127.1.170.148:39417
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.170.190:39413
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260923 20:10:16.536028 11041 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:10:16.536355 11041 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:10:16.536438 11041 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:10:16.544301 11041 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 20:10:16.544443 11041 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:10:16.544481 11041 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:10:16.544530 11041 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 20:10:16.544575 11041 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 20:10:16.554193 11041 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.170.148:39417
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790194121577027-1706-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790194121577027-1706-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.170.190:39413
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.170.190:39413
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790194121577027-1706-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.170.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790194121577027-1706-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:10:16.555580 11041 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:10:16.557338 11041 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:10:16.570106 11046 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:10:16.570024 11047 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:10:16.570075 11049 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:10:16.571592 11041 server_base.cc:1034] running on GCE node
I20260923 20:10:16.572368 11041 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:10:16.573853 11041 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:10:16.575157 11041 hybrid_clock.cc:648] HybridClock initialized: now 1790194216575121 us; error 44 us; skew 500 ppm
I20260923 20:10:16.575524 11041 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260923 20:10:16.578234 11041 webserver.cc:460] Webserver started at http://127.1.170.190:36955/ using document root <none> and password file <none>
I20260923 20:10:16.578958 11041 fs_manager.cc:359] Metadata directory not provided
I20260923 20:10:16.579089 11041 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:10:16.579452 11041 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:10:16.582435 11041 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790194121577027-1706-0/minicluster-data/master-0/data/instance:
uuid: "abd7799b2d4b465eb4110e5dbf7fe39c"
format_stamp: "Formatted at 2026-09-23 20:10:16 on dist-test-slave-th4g"
server_key: "13f7a7655d1282f6409299fe4f239612"
server_key_iv: "21e9080912fbfecb6645a3c734b67d82"
server_key_version: "encryptionkey@0"
I20260923 20:10:16.583287 11041 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790194121577027-1706-0/minicluster-data/master-0/wal/instance:
uuid: "abd7799b2d4b465eb4110e5dbf7fe39c"
format_stamp: "Formatted at 2026-09-23 20:10:16 on dist-test-slave-th4g"
server_key: "13f7a7655d1282f6409299fe4f239612"
server_key_iv: "21e9080912fbfecb6645a3c734b67d82"
server_key_version: "encryptionkey@0"
I20260923 20:10:16.588546 11041 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260923 20:10:16.592763 11055 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:10:16.594420 11041 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.002s
I20260923 20:10:16.594621 11041 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790194121577027-1706-0/minicluster-data/master-0/data,/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790194121577027-1706-0/minicluster-data/master-0/wal
uuid: "abd7799b2d4b465eb4110e5dbf7fe39c"
format_stamp: "Formatted at 2026-09-23 20:10:16 on dist-test-slave-th4g"
server_key: "13f7a7655d1282f6409299fe4f239612"
server_key_iv: "21e9080912fbfecb6645a3c734b67d82"
server_key_version: "encryptionkey@0"
I20260923 20:10:16.594866 11041 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790194121577027-1706-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790194121577027-1706-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790194121577027-1706-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:10:16.616072 11041 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:10:16.616992 11041 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:10:16.617316 11041 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:10:16.636375 11041 rpc_server.cc:307] RPC server started. Bound to: 127.1.170.190:39413
I20260923 20:10:16.636400 11106 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.170.190:39413 every 8 connection(s)
I20260923 20:10:16.637780 11041 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790194121577027-1706-0/minicluster-data/master-0/data/info.pb
I20260923 20:10:16.641727 11107 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:10:16.644375 1706 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqUuku6/build/asan/bin/kudu as pid 11041
I20260923 20:10:16.644586 1706 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790194121577027-1706-0/minicluster-data/master-0/wal/instance
I20260923 20:10:16.645378 1706 external_mini_cluster.cc:1428] Setting key 39dd8d4f7738a8dc6ab8b3d46509bc38
I20260923 20:10:16.653784 11107 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P abd7799b2d4b465eb4110e5dbf7fe39c: Bootstrap starting.
I20260923 20:10:16.657044 11107 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P abd7799b2d4b465eb4110e5dbf7fe39c: Neither blocks nor log segments found. Creating new log.
I20260923 20:10:16.658206 11107 log.cc:824] T 00000000000000000000000000000000 P abd7799b2d4b465eb4110e5dbf7fe39c: Log is configured to *not* fsync() on all Append() calls
I20260923 20:10:16.659198 11110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:10:16.647592 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42492 (local address 127.1.170.190:39413)
0923 20:10:16.648589 (+ 997us) server_negotiation.cc:206] Beginning negotiation
0923 20:10:16.648600 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0923 20:10:16.648640 (+ 40us) server_negotiation.cc:407] Connection header received
0923 20:10:16.650361 (+ 1721us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:10:16.650384 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:10:16.650849 (+ 465us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:10:16.651253 (+ 404us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:10:16.651851 (+ 598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:10:16.653044 (+ 1193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:10:16.654606 (+ 1562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:10:16.655282 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:10:16.655508 (+ 226us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:10:16.656648 (+ 1140us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:10:16.656656 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:10:16.656669 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 20:10:16.657221 (+ 552us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:10:16.657335 (+ 114us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 20:10:16.657360 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:10:16.657498 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0923 20:10:16.657719 (+ 221us) server_negotiation.cc:299] Negotiation successful
0923 20:10:16.658048 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":574,"thread_start_us":339,"threads_started":1}
I20260923 20:10:16.661284 11107 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P abd7799b2d4b465eb4110e5dbf7fe39c: No bootstrap required, opened a new log
I20260923 20:10:16.669322 11107 raft_consensus.cc:348] T 00000000000000000000000000000000 P abd7799b2d4b465eb4110e5dbf7fe39c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abd7799b2d4b465eb4110e5dbf7fe39c" member_type: VOTER last_known_addr { host: "127.1.170.190" port: 39413 } }
I20260923 20:10:16.669691 11107 raft_consensus.cc:374] T 00000000000000000000000000000000 P abd7799b2d4b465eb4110e5dbf7fe39c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:10:16.669770 11107 raft_consensus.cc:729] T 00000000000000000000000000000000 P abd7799b2d4b465eb4110e5dbf7fe39c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: abd7799b2d4b465eb4110e5dbf7fe39c, State: Initialized, Role: FOLLOWER
I20260923 20:10:16.670375 11107 consensus_queue.cc:260] T 00000000000000000000000000000000 P abd7799b2d4b465eb4110e5dbf7fe39c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abd7799b2d4b465eb4110e5dbf7fe39c" member_type: VOTER last_known_addr { host: "127.1.170.190" port: 39413 } }
I20260923 20:10:16.670583 11107 raft_consensus.cc:388] T 00000000000000000000000000000000 P abd7799b2d4b465eb4110e5dbf7fe39c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 20:10:16.670674 11107 raft_consensus.cc:482] T 00000000000000000000000000000000 P abd7799b2d4b465eb4110e5dbf7fe39c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 20:10:16.670807 11107 raft_consensus.cc:3037] T 00000000000000000000000000000000 P abd7799b2d4b465eb4110e5dbf7fe39c [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:10:16.672986 11107 raft_consensus.cc:504] T 00000000000000000000000000000000 P abd7799b2d4b465eb4110e5dbf7fe39c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abd7799b2d4b465eb4110e5dbf7fe39c" member_type: VOTER last_known_addr { host: "127.1.170.190" port: 39413 } }
I20260923 20:10:16.673647 11107 leader_election.cc:302] T 00000000000000000000000000000000 P abd7799b2d4b465eb4110e5dbf7fe39c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: abd7799b2d4b465eb4110e5dbf7fe39c; no voters:
I20260923 20:10:16.674451 11107 leader_election.cc:288] T 00000000000000000000000000000000 P abd7799b2d4b465eb4110e5dbf7fe39c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 20:10:16.674726 11112 raft_consensus.cc:2781] T 00000000000000000000000000000000 P abd7799b2d4b465eb4110e5dbf7fe39c [term 1 FOLLOWER]: Leader election won for term 1
I20260923 20:10:16.675958 11112 raft_consensus.cc:686] T 00000000000000000000000000000000 P abd7799b2d4b465eb4110e5dbf7fe39c [term 1 LEADER]: Becoming Leader. State: Replica: abd7799b2d4b465eb4110e5dbf7fe39c, State: Running, Role: LEADER
I20260923 20:10:16.676535 11112 consensus_queue.cc:237] T 00000000000000000000000000000000 P abd7799b2d4b465eb4110e5dbf7fe39c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abd7799b2d4b465eb4110e5dbf7fe39c" member_type: VOTER last_known_addr { host: "127.1.170.190" port: 39413 } }
I20260923 20:10:16.676985 11107 sys_catalog.cc:564] T 00000000000000000000000000000000 P abd7799b2d4b465eb4110e5dbf7fe39c [sys.catalog]: configured and running, proceeding with master startup.
I20260923 20:10:16.679363 11114 sys_catalog.cc:455] T 00000000000000000000000000000000 P abd7799b2d4b465eb4110e5dbf7fe39c [sys.catalog]: SysCatalogTable state changed. Reason: New leader abd7799b2d4b465eb4110e5dbf7fe39c. Latest consensus state: current_term: 1 leader_uuid: "abd7799b2d4b465eb4110e5dbf7fe39c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abd7799b2d4b465eb4110e5dbf7fe39c" member_type: VOTER last_known_addr { host: "127.1.170.190" port: 39413 } } }
I20260923 20:10:16.679419 11113 sys_catalog.cc:455] T 00000000000000000000000000000000 P abd7799b2d4b465eb4110e5dbf7fe39c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "abd7799b2d4b465eb4110e5dbf7fe39c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abd7799b2d4b465eb4110e5dbf7fe39c" member_type: VOTER last_known_addr { host: "127.1.170.190" port: 39413 } } }
I20260923 20:10:16.679756 11114 sys_catalog.cc:458] T 00000000000000000000000000000000 P abd7799b2d4b465eb4110e5dbf7fe39c [sys.catalog]: This master's current role is: LEADER
I20260923 20:10:16.679833 11113 sys_catalog.cc:458] T 00000000000000000000000000000000 P abd7799b2d4b465eb4110e5dbf7fe39c [sys.catalog]: This master's current role is: LEADER
I20260923 20:10:16.683388 11119 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 20:10:16.689072 11119 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 20:10:16.697794 11119 catalog_manager.cc:1347] Generated new cluster ID: bae5df99efce45e896ac57fea9aa8c1a
I20260923 20:10:16.697978 11119 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 20:10:16.715238 11119 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 20:10:16.716548 11119 catalog_manager.cc:1504] Loading token signing keys...
I20260923 20:10:16.726543 11119 catalog_manager.cc:5868] T 00000000000000000000000000000000 P abd7799b2d4b465eb4110e5dbf7fe39c: Generated new TSK 0
I20260923 20:10:16.727481 11119 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 20:10:16.735160 1706 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqUuku6/build/asan/bin/kudu
/tmp/dist-test-taskqUuku6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790194121577027-1706-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790194121577027-1706-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790194121577027-1706-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790194121577027-1706-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.170.129:0
--local_ip_for_outbound_sockets=127.1.170.129
--webserver_interface=127.1.170.129
--webserver_port=0
--tserver_master_addrs=127.1.170.190:39413
--builtin_ntp_servers=127.1.170.148:39417
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260923 20:10:17.074805 11131 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:10:17.075096 11131 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:10:17.075239 11131 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:10:17.083252 11131 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:10:17.083403 11131 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:10:17.083580 11131 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.170.129
I20260923 20:10:17.093456 11131 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.170.148:39417
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790194121577027-1706-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790194121577027-1706-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.170.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790194121577027-1706-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.170.129
--webserver_port=0
--tserver_master_addrs=127.1.170.190:39413
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.170.129
--log_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790194121577027-1706-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:10:17.094806 11131 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:10:17.096570 11131 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:10:17.109833 11136 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:10:17.109760 11137 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:10:17.109792 11139 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:10:17.110996 11131 server_base.cc:1034] running on GCE node
I20260923 20:10:17.111647 11131 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:10:17.112802 11131 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:10:17.114058 11131 hybrid_clock.cc:648] HybridClock initialized: now 1790194217113995 us; error 82 us; skew 500 ppm
I20260923 20:10:17.114419 11131 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260923 20:10:17.117053 11131 webserver.cc:460] Webserver started at http://127.1.170.129:44433/ using document root <none> and password file <none>
I20260923 20:10:17.117777 11131 fs_manager.cc:359] Metadata directory not provided
I20260923 20:10:17.117923 11131 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:10:17.118255 11131 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:10:17.121343 11131 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790194121577027-1706-0/minicluster-data/ts-0/data/instance:
uuid: "efb8dcb4be7a40aa961685fd176dc3d0"
format_stamp: "Formatted at 2026-09-23 20:10:17 on dist-test-slave-th4g"
server_key: "e40dea678b16345bbdf7bb14ca75ded2"
server_key_iv: "b955d9654bd812d1a4658578c5621447"
server_key_version: "encryptionkey@0"
I20260923 20:10:17.122262 11131 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790194121577027-1706-0/minicluster-data/ts-0/wal/instance:
uuid: "efb8dcb4be7a40aa961685fd176dc3d0"
format_stamp: "Formatted at 2026-09-23 20:10:17 on dist-test-slave-th4g"
server_key: "e40dea678b16345bbdf7bb14ca75ded2"
server_key_iv: "b955d9654bd812d1a4658578c5621447"
server_key_version: "encryptionkey@0"
I20260923 20:10:17.127384 11131 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260923 20:10:17.130798 11145 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:10:17.132172 11131 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260923 20:10:17.132393 11131 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790194121577027-1706-0/minicluster-data/ts-0/data,/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790194121577027-1706-0/minicluster-data/ts-0/wal
uuid: "efb8dcb4be7a40aa961685fd176dc3d0"
format_stamp: "Formatted at 2026-09-23 20:10:17 on dist-test-slave-th4g"
server_key: "e40dea678b16345bbdf7bb14ca75ded2"
server_key_iv: "b955d9654bd812d1a4658578c5621447"
server_key_version: "encryptionkey@0"
I20260923 20:10:17.132615 11131 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790194121577027-1706-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790194121577027-1706-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790194121577027-1706-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:10:17.147838 11131 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:10:17.148686 11131 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:10:17.148991 11131 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:10:17.149998 11131 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 20:10:17.151437 11131 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 20:10:17.151571 11131 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 20:10:17.151700 11131 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 20:10:17.151787 11131 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 20:10:17.186157 11131 rpc_server.cc:307] RPC server started. Bound to: 127.1.170.129:37747
I20260923 20:10:17.186223 11257 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.170.129:37747 every 8 connection(s)
I20260923 20:10:17.187493 11131 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790194121577027-1706-0/minicluster-data/ts-0/data/info.pb
I20260923 20:10:17.191363 1706 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqUuku6/build/asan/bin/kudu as pid 11131
I20260923 20:10:17.191581 1706 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790194121577027-1706-0/minicluster-data/ts-0/wal/instance
I20260923 20:10:17.192269 1706 external_mini_cluster.cc:1428] Setting key ce27c04da13c1e7197dd913ee05ff4f8
I20260923 20:10:17.195209 1706 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqUuku6/build/asan/bin/kudu
/tmp/dist-test-taskqUuku6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790194121577027-1706-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790194121577027-1706-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790194121577027-1706-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790194121577027-1706-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.170.130:0
--local_ip_for_outbound_sockets=127.1.170.130
--webserver_interface=127.1.170.130
--webserver_port=0
--tserver_master_addrs=127.1.170.190:39413
--builtin_ntp_servers=127.1.170.148:39417
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260923 20:10:17.202219 11260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:10:17.190251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.170.129:59697 (local address 127.1.170.190:39413)
0923 20:10:17.190635 (+ 384us) server_negotiation.cc:206] Beginning negotiation
0923 20:10:17.190639 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 20:10:17.192028 (+ 1389us) server_negotiation.cc:407] Connection header received
0923 20:10:17.193978 (+ 1950us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:10:17.193984 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:10:17.194079 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:10:17.194205 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:10:17.195611 (+ 1406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:10:17.196346 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:10:17.199051 (+ 2705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:10:17.199534 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:10:17.199647 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:10:17.201358 (+ 1711us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:10:17.201363 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:10:17.201367 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 20:10:17.201388 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:10:17.201430 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 20:10:17.201442 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:10:17.201519 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0923 20:10:17.201799 (+ 280us) server_negotiation.cc:299] Negotiation successful
0923 20:10:17.201933 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":279,"thread_start_us":194,"threads_started":1}
I20260923 20:10:17.203343 11261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:10:17.190813 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.190:39413 (local address 127.1.170.129:59697)
0923 20:10:17.191844 (+ 1031us) negotiation.cc:107] Waiting for socket to connect
0923 20:10:17.191894 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0923 20:10:17.193641 (+ 1747us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:10:17.194466 (+ 825us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:10:17.194476 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:10:17.194594 (+ 118us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:10:17.195429 (+ 835us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:10:17.195458 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:10:17.196498 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:10:17.196507 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:10:17.198835 (+ 2328us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:10:17.198855 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:10:17.199797 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:10:17.199807 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:10:17.200143 (+ 336us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:10:17.200984 (+ 841us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 20:10:17.200990 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:10:17.201047 (+ 57us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 20:10:17.201155 (+ 108us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:10:17.201544 (+ 389us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:10:17.201557 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:10:17.201568 (+ 11us) client_negotiation.cc:769] Sending connection context
0923 20:10:17.201766 (+ 198us) client_negotiation.cc:240] Negotiation successful
0923 20:10:17.202243 (+ 477us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":519,"thread_start_us":352,"threads_started":1}
I20260923 20:10:17.204972 11258 heartbeater.cc:347] Connected to a master server at 127.1.170.190:39413
I20260923 20:10:17.205329 11258 heartbeater.cc:464] Registering TS with master...
I20260923 20:10:17.206190 11258 heartbeater.cc:511] Master 127.1.170.190:39413 requested a full tablet report, sending...
I20260923 20:10:17.208282 11072 ts_manager.cc:194] Registered new tserver with Master: efb8dcb4be7a40aa961685fd176dc3d0 (127.1.170.129:37747)
I20260923 20:10:17.209761 11072 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.170.129:59697
W20260923 20:10:17.528891 11262 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:10:17.529217 11262 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:10:17.529361 11262 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:10:17.536640 11262 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:10:17.536777 11262 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:10:17.536942 11262 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.170.130
I20260923 20:10:17.546129 11262 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.170.148:39417
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790194121577027-1706-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790194121577027-1706-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.170.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790194121577027-1706-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.170.130
--webserver_port=0
--tserver_master_addrs=127.1.170.190:39413
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.170.130
--log_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790194121577027-1706-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:10:17.547501 11262 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:10:17.549363 11262 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:10:17.565207 11268 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:10:17.565330 11270 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:10:17.565945 11267 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:10:17.566449 11262 server_base.cc:1034] running on GCE node
I20260923 20:10:17.567209 11262 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:10:17.568470 11262 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:10:17.569708 11262 hybrid_clock.cc:648] HybridClock initialized: now 1790194217569637 us; error 59 us; skew 500 ppm
I20260923 20:10:17.570125 11262 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260923 20:10:17.572815 11262 webserver.cc:460] Webserver started at http://127.1.170.130:37031/ using document root <none> and password file <none>
I20260923 20:10:17.573730 11262 fs_manager.cc:359] Metadata directory not provided
I20260923 20:10:17.573856 11262 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:10:17.574276 11262 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:10:17.578608 11262 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790194121577027-1706-0/minicluster-data/ts-1/data/instance:
uuid: "1059d548d7f94243b84e9d7e257ed427"
format_stamp: "Formatted at 2026-09-23 20:10:17 on dist-test-slave-th4g"
server_key: "18fcf773c881a2eaacb83f394ea70391"
server_key_iv: "1a71e1fde5b80d572944bfc8a0f8b703"
server_key_version: "encryptionkey@0"
I20260923 20:10:17.579757 11262 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790194121577027-1706-0/minicluster-data/ts-1/wal/instance:
uuid: "1059d548d7f94243b84e9d7e257ed427"
format_stamp: "Formatted at 2026-09-23 20:10:17 on dist-test-slave-th4g"
server_key: "18fcf773c881a2eaacb83f394ea70391"
server_key_iv: "1a71e1fde5b80d572944bfc8a0f8b703"
server_key_version: "encryptionkey@0"
I20260923 20:10:17.587226 11262 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.006s sys 0.001s
I20260923 20:10:17.592065 11276 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:10:17.593853 11262 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260923 20:10:17.594058 11262 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790194121577027-1706-0/minicluster-data/ts-1/data,/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790194121577027-1706-0/minicluster-data/ts-1/wal
uuid: "1059d548d7f94243b84e9d7e257ed427"
format_stamp: "Formatted at 2026-09-23 20:10:17 on dist-test-slave-th4g"
server_key: "18fcf773c881a2eaacb83f394ea70391"
server_key_iv: "1a71e1fde5b80d572944bfc8a0f8b703"
server_key_version: "encryptionkey@0"
I20260923 20:10:17.594331 11262 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790194121577027-1706-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790194121577027-1706-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790194121577027-1706-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:10:17.617743 11262 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:10:17.618810 11262 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:10:17.619062 11262 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:10:17.619946 11262 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 20:10:17.621340 11262 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 20:10:17.621449 11262 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 20:10:17.621559 11262 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 20:10:17.621632 11262 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 20:10:17.659793 11262 rpc_server.cc:307] RPC server started. Bound to: 127.1.170.130:34533
I20260923 20:10:17.659888 11388 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.170.130:34533 every 8 connection(s)
I20260923 20:10:17.661042 11262 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790194121577027-1706-0/minicluster-data/ts-1/data/info.pb
I20260923 20:10:17.671361 1706 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqUuku6/build/asan/bin/kudu as pid 11262
I20260923 20:10:17.671550 1706 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790194121577027-1706-0/minicluster-data/ts-1/wal/instance
I20260923 20:10:17.671991 1706 external_mini_cluster.cc:1428] Setting key 32d6dd59e2ab88c086921513648d29bb
I20260923 20:10:17.673120 11260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:10:17.663533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.170.130:33367 (local address 127.1.170.190:39413)
0923 20:10:17.663714 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0923 20:10:17.663719 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 20:10:17.665063 (+ 1344us) server_negotiation.cc:407] Connection header received
0923 20:10:17.666836 (+ 1773us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:10:17.666842 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:10:17.666930 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:10:17.667048 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:10:17.668064 (+ 1016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:10:17.668766 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:10:17.670362 (+ 1596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:10:17.670749 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:10:17.670886 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:10:17.672277 (+ 1391us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:10:17.672283 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:10:17.672288 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 20:10:17.672315 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:10:17.672344 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 20:10:17.672355 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:10:17.672450 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0923 20:10:17.672680 (+ 230us) server_negotiation.cc:299] Negotiation successful
0923 20:10:17.672773 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260923 20:10:17.674201 11391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:10:17.664061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.190:39413 (local address 127.1.170.130:33367)
0923 20:10:17.664904 (+ 843us) negotiation.cc:107] Waiting for socket to connect
0923 20:10:17.664938 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0923 20:10:17.666573 (+ 1635us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:10:17.667283 (+ 710us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:10:17.667290 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:10:17.667369 (+ 79us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:10:17.667895 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:10:17.667910 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:10:17.668919 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:10:17.668924 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:10:17.670223 (+ 1299us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:10:17.670235 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:10:17.670967 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:10:17.670974 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:10:17.671206 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:10:17.671944 (+ 738us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 20:10:17.671946 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:10:17.671984 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 20:10:17.672054 (+ 70us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:10:17.672448 (+ 394us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:10:17.672459 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:10:17.672470 (+ 11us) client_negotiation.cc:769] Sending connection context
0923 20:10:17.672658 (+ 188us) client_negotiation.cc:240] Negotiation successful
0923 20:10:17.673035 (+ 377us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":446,"thread_start_us":265,"threads_started":1}
I20260923 20:10:17.674784 1706 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqUuku6/build/asan/bin/kudu
/tmp/dist-test-taskqUuku6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790194121577027-1706-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790194121577027-1706-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790194121577027-1706-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790194121577027-1706-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.170.131:0
--local_ip_for_outbound_sockets=127.1.170.131
--webserver_interface=127.1.170.131
--webserver_port=0
--tserver_master_addrs=127.1.170.190:39413
--builtin_ntp_servers=127.1.170.148:39417
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260923 20:10:17.675540 11389 heartbeater.cc:347] Connected to a master server at 127.1.170.190:39413
I20260923 20:10:17.675845 11389 heartbeater.cc:464] Registering TS with master...
I20260923 20:10:17.676734 11389 heartbeater.cc:511] Master 127.1.170.190:39413 requested a full tablet report, sending...
I20260923 20:10:17.678488 11072 ts_manager.cc:194] Registered new tserver with Master: 1059d548d7f94243b84e9d7e257ed427 (127.1.170.130:34533)
I20260923 20:10:17.679292 11072 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.170.130:33367
W20260923 20:10:17.998649 11392 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:10:17.998970 11392 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:10:17.999132 11392 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:10:18.006462 11392 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:10:18.006552 11392 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:10:18.006650 11392 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.170.131
I20260923 20:10:18.015420 11392 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.170.148:39417
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790194121577027-1706-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790194121577027-1706-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.170.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790194121577027-1706-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.170.131
--webserver_port=0
--tserver_master_addrs=127.1.170.190:39413
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.170.131
--log_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790194121577027-1706-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:10:18.016598 11392 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:10:18.018239 11392 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:10:18.031224 11398 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:10:18.031275 11397 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:10:18.031105 11400 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:10:18.031926 11392 server_base.cc:1034] running on GCE node
I20260923 20:10:18.032555 11392 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:10:18.033509 11392 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:10:18.034739 11392 hybrid_clock.cc:648] HybridClock initialized: now 1790194218034699 us; error 54 us; skew 500 ppm
I20260923 20:10:18.035060 11392 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260923 20:10:18.041764 11392 webserver.cc:460] Webserver started at http://127.1.170.131:46811/ using document root <none> and password file <none>
I20260923 20:10:18.042533 11392 fs_manager.cc:359] Metadata directory not provided
I20260923 20:10:18.042678 11392 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:10:18.043007 11392 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:10:18.046022 11392 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790194121577027-1706-0/minicluster-data/ts-2/data/instance:
uuid: "24698b842cb04aaa93210affc8d2210e"
format_stamp: "Formatted at 2026-09-23 20:10:18 on dist-test-slave-th4g"
server_key: "85d9c6a22544f01fc03f26e8b4bdcb40"
server_key_iv: "a1effdbeb9a966f44f0388119a1c03c9"
server_key_version: "encryptionkey@0"
I20260923 20:10:18.046890 11392 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790194121577027-1706-0/minicluster-data/ts-2/wal/instance:
uuid: "24698b842cb04aaa93210affc8d2210e"
format_stamp: "Formatted at 2026-09-23 20:10:18 on dist-test-slave-th4g"
server_key: "85d9c6a22544f01fc03f26e8b4bdcb40"
server_key_iv: "a1effdbeb9a966f44f0388119a1c03c9"
server_key_version: "encryptionkey@0"
I20260923 20:10:18.052111 11392 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.005s
I20260923 20:10:18.055707 11406 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:10:18.057112 11392 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20260923 20:10:18.057298 11392 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790194121577027-1706-0/minicluster-data/ts-2/data,/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790194121577027-1706-0/minicluster-data/ts-2/wal
uuid: "24698b842cb04aaa93210affc8d2210e"
format_stamp: "Formatted at 2026-09-23 20:10:18 on dist-test-slave-th4g"
server_key: "85d9c6a22544f01fc03f26e8b4bdcb40"
server_key_iv: "a1effdbeb9a966f44f0388119a1c03c9"
server_key_version: "encryptionkey@0"
I20260923 20:10:18.057518 11392 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790194121577027-1706-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790194121577027-1706-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790194121577027-1706-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:10:18.098482 11392 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:10:18.099602 11392 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:10:18.099931 11392 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:10:18.101035 11392 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 20:10:18.102643 11392 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 20:10:18.102773 11392 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 20:10:18.102917 11392 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 20:10:18.103013 11392 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 20:10:18.138063 11392 rpc_server.cc:307] RPC server started. Bound to: 127.1.170.131:37929
I20260923 20:10:18.138101 11518 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.170.131:37929 every 8 connection(s)
I20260923 20:10:18.139432 11392 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790194121577027-1706-0/minicluster-data/ts-2/data/info.pb
I20260923 20:10:18.149353 1706 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqUuku6/build/asan/bin/kudu as pid 11392
I20260923 20:10:18.149542 1706 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790194121577027-1706-0/minicluster-data/ts-2/wal/instance
I20260923 20:10:18.150082 1706 external_mini_cluster.cc:1428] Setting key aff3ec880f6eda35ea150cc29e97e16a
I20260923 20:10:18.152009 11260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:10:18.142068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.170.131:52201 (local address 127.1.170.190:39413)
0923 20:10:18.142243 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0923 20:10:18.142248 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 20:10:18.143609 (+ 1361us) server_negotiation.cc:407] Connection header received
0923 20:10:18.145329 (+ 1720us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:10:18.145335 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:10:18.145425 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:10:18.145537 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:10:18.146656 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:10:18.147400 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:10:18.148932 (+ 1532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:10:18.149551 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:10:18.149691 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:10:18.151159 (+ 1468us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:10:18.151165 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:10:18.151171 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 20:10:18.151195 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:10:18.151233 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 20:10:18.151248 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:10:18.151336 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0923 20:10:18.151591 (+ 255us) server_negotiation.cc:299] Negotiation successful
0923 20:10:18.151661 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260923 20:10:18.153303 11521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:10:18.142609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.190:39413 (local address 127.1.170.131:52201)
0923 20:10:18.143460 (+ 851us) negotiation.cc:107] Waiting for socket to connect
0923 20:10:18.143498 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0923 20:10:18.145054 (+ 1556us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:10:18.145759 (+ 705us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:10:18.145767 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:10:18.145852 (+ 85us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:10:18.146499 (+ 647us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:10:18.146515 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:10:18.147566 (+ 1051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:10:18.147578 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:10:18.148786 (+ 1208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:10:18.148797 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:10:18.149792 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:10:18.149797 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:10:18.150053 (+ 256us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:10:18.150804 (+ 751us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 20:10:18.150807 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:10:18.150846 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 20:10:18.150946 (+ 100us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:10:18.151358 (+ 412us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:10:18.151369 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:10:18.151377 (+ 8us) client_negotiation.cc:769] Sending connection context
0923 20:10:18.151570 (+ 193us) client_negotiation.cc:240] Negotiation successful
0923 20:10:18.151969 (+ 399us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":447,"thread_start_us":277,"threads_started":1}
I20260923 20:10:18.154776 11519 heartbeater.cc:347] Connected to a master server at 127.1.170.190:39413
I20260923 20:10:18.155074 11519 heartbeater.cc:464] Registering TS with master...
I20260923 20:10:18.155855 11519 heartbeater.cc:511] Master 127.1.170.190:39413 requested a full tablet report, sending...
I20260923 20:10:18.157191 11072 ts_manager.cc:194] Registered new tserver with Master: 24698b842cb04aaa93210affc8d2210e (127.1.170.131:37929)
I20260923 20:10:18.158202 11072 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.170.131:52201
I20260923 20:10:18.165513 1706 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260923 20:10:18.172567 11260 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 20:10:18.170679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42502 (local address 127.1.170.190:39413)
0923 20:10:18.170857 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0923 20:10:18.170861 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 20:10:18.171422 (+ 561us) server_negotiation.cc:407] Connection header received
0923 20:10:18.171609 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:10:18.171613 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:10:18.171692 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:10:18.171782 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:10:18.172306 (+ 524us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42502: BlockingRecv error: recv got EOF from 127.0.0.1:42502 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
I20260923 20:10:18.173249 1706 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskqUuku6/build/asan/bin/kudu pid 11131
I20260923 20:10:18.184592 11528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:10:18.174254 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39512 (local address 127.1.170.129:37747)
0923 20:10:18.174875 (+ 621us) server_negotiation.cc:206] Beginning negotiation
0923 20:10:18.174883 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 20:10:18.177666 (+ 2783us) server_negotiation.cc:407] Connection header received
0923 20:10:18.177864 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:10:18.177898 (+ 34us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:10:18.178148 (+ 250us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:10:18.178325 (+ 177us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:10:18.178769 (+ 444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:10:18.179886 (+ 1117us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:10:18.181180 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:10:18.181783 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:10:18.181958 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:10:18.183081 (+ 1123us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:10:18.183096 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:10:18.183110 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 20:10:18.183731 (+ 621us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:10:18.183791 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 20:10:18.183812 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:10:18.183916 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0923 20:10:18.184051 (+ 135us) server_negotiation.cc:299] Negotiation successful
0923 20:10:18.184198 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":421,"thread_start_us":244,"threads_started":1}
I20260923 20:10:18.186077 11253 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:10:18.268311 11258 heartbeater.cc:503] Master 127.1.170.190:39413 was elected leader, sending a full tablet report...
I20260923 20:10:18.269604 1706 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqUuku6/build/asan/bin/kudu with pid 11131
W20260923 20:10:18.286463 11058 connection.cc:535] server connection from 127.1.170.129:59697 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260923 20:10:18.286814 1706 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskqUuku6/build/asan/bin/kudu pid 11262
I20260923 20:10:18.294533 11530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:10:18.287712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46442 (local address 127.1.170.130:34533)
0923 20:10:18.288190 (+ 478us) server_negotiation.cc:206] Beginning negotiation
0923 20:10:18.288197 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:10:18.288221 (+ 24us) server_negotiation.cc:407] Connection header received
0923 20:10:18.288326 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:10:18.288333 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:10:18.288540 (+ 207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:10:18.288716 (+ 176us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:10:18.289089 (+ 373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:10:18.290175 (+ 1086us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:10:18.291411 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:10:18.291937 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:10:18.292073 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:10:18.293131 (+ 1058us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:10:18.293138 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:10:18.293150 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 20:10:18.293703 (+ 553us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:10:18.293752 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 20:10:18.293771 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:10:18.293912 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0923 20:10:18.294073 (+ 161us) server_negotiation.cc:299] Negotiation successful
0923 20:10:18.294208 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":363,"thread_start_us":252,"threads_started":1}
I20260923 20:10:18.295727 11384 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:10:18.377202 1706 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqUuku6/build/asan/bin/kudu with pid 11262
I20260923 20:10:18.393599 1706 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskqUuku6/build/asan/bin/kudu pid 11392
I20260923 20:10:18.401867 11532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:10:18.394705 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46018 (local address 127.1.170.131:37929)
0923 20:10:18.395338 (+ 633us) server_negotiation.cc:206] Beginning negotiation
0923 20:10:18.395345 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:10:18.395374 (+ 29us) server_negotiation.cc:407] Connection header received
0923 20:10:18.395430 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:10:18.395436 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:10:18.395664 (+ 228us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:10:18.395799 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:10:18.396216 (+ 417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:10:18.397313 (+ 1097us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:10:18.398776 (+ 1463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:10:18.399369 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:10:18.399479 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:10:18.400488 (+ 1009us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:10:18.400497 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:10:18.400509 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 20:10:18.401074 (+ 565us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:10:18.401117 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 20:10:18.401137 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:10:18.401231 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0923 20:10:18.401366 (+ 135us) server_negotiation.cc:299] Negotiation successful
0923 20:10:18.401501 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":497,"thread_start_us":343,"threads_started":1}
I20260923 20:10:18.403108 11514 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:10:18.486428 1706 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqUuku6/build/asan/bin/kudu with pid 11392
I20260923 20:10:18.502890 1706 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskqUuku6/build/asan/bin/kudu pid 11041
I20260923 20:10:18.504243 11102 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:10:18.605327 1706 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqUuku6/build/asan/bin/kudu with pid 11041
2026-09-23T20:10:18Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (2508 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-09-23T20:10:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T20:10:18Z Disabled control of system clock
I20260923 20:10:18.675920 1706 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqUuku6/build/asan/bin/kudu
/tmp/dist-test-taskqUuku6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790194121577027-1706-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790194121577027-1706-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790194121577027-1706-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790194121577027-1706-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.170.190:45817
--webserver_interface=127.1.170.190
--webserver_port=0
--builtin_ntp_servers=127.1.170.148:42537
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.170.190:45817
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260923 20:10:19.003643 11538 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:10:19.003955 11538 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:10:19.004105 11538 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:10:19.011770 11538 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 20:10:19.011900 11538 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:10:19.011978 11538 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:10:19.012075 11538 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 20:10:19.012166 11538 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 20:10:19.021576 11538 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.170.148:42537
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790194121577027-1706-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790194121577027-1706-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.170.190:45817
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.170.190:45817
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790194121577027-1706-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.170.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790194121577027-1706-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:10:19.022939 11538 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:10:19.024698 11538 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:10:19.037001 11543 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:10:19.037052 11546 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:10:19.037075 11544 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:10:19.038348 11538 server_base.cc:1034] running on GCE node
I20260923 20:10:19.039136 11538 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:10:19.040781 11538 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:10:19.042246 11538 hybrid_clock.cc:648] HybridClock initialized: now 1790194219042167 us; error 76 us; skew 500 ppm
I20260923 20:10:19.042711 11538 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260923 20:10:19.045488 11538 webserver.cc:460] Webserver started at http://127.1.170.190:40169/ using document root <none> and password file <none>
I20260923 20:10:19.046346 11538 fs_manager.cc:359] Metadata directory not provided
I20260923 20:10:19.046484 11538 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:10:19.046821 11538 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:10:19.051362 11538 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790194121577027-1706-0/minicluster-data/master-0/data/instance:
uuid: "5627a53cb9e7401695a394805cda654a"
format_stamp: "Formatted at 2026-09-23 20:10:19 on dist-test-slave-th4g"
server_key: "cdc90e8a45140375ba75a26ca902ae13"
server_key_iv: "80ea6db0f5a1f788771a8ea200dd1daf"
server_key_version: "encryptionkey@0"
I20260923 20:10:19.052206 11538 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790194121577027-1706-0/minicluster-data/master-0/wal/instance:
uuid: "5627a53cb9e7401695a394805cda654a"
format_stamp: "Formatted at 2026-09-23 20:10:19 on dist-test-slave-th4g"
server_key: "cdc90e8a45140375ba75a26ca902ae13"
server_key_iv: "80ea6db0f5a1f788771a8ea200dd1daf"
server_key_version: "encryptionkey@0"
I20260923 20:10:19.057629 11538 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260923 20:10:19.061458 11552 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:10:19.063536 11538 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260923 20:10:19.063774 11538 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790194121577027-1706-0/minicluster-data/master-0/data,/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790194121577027-1706-0/minicluster-data/master-0/wal
uuid: "5627a53cb9e7401695a394805cda654a"
format_stamp: "Formatted at 2026-09-23 20:10:19 on dist-test-slave-th4g"
server_key: "cdc90e8a45140375ba75a26ca902ae13"
server_key_iv: "80ea6db0f5a1f788771a8ea200dd1daf"
server_key_version: "encryptionkey@0"
I20260923 20:10:19.064026 11538 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790194121577027-1706-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790194121577027-1706-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790194121577027-1706-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:10:19.070183 11538 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:10:19.071017 11538 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:10:19.071333 11538 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:10:19.091907 11538 rpc_server.cc:307] RPC server started. Bound to: 127.1.170.190:45817
I20260923 20:10:19.091939 11603 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.170.190:45817 every 8 connection(s)
I20260923 20:10:19.093333 11538 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790194121577027-1706-0/minicluster-data/master-0/data/info.pb
I20260923 20:10:19.096715 1706 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqUuku6/build/asan/bin/kudu as pid 11538
I20260923 20:10:19.096936 1706 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790194121577027-1706-0/minicluster-data/master-0/wal/instance
I20260923 20:10:19.097281 11604 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:10:19.097618 1706 external_mini_cluster.cc:1428] Setting key e7e324a06f3e295f905f884683288439
I20260923 20:10:19.107636 11607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:10:19.100052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50786 (local address 127.1.170.190:45817)
0923 20:10:19.101087 (+ 1035us) server_negotiation.cc:206] Beginning negotiation
0923 20:10:19.101098 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0923 20:10:19.101141 (+ 43us) server_negotiation.cc:407] Connection header received
0923 20:10:19.103601 (+ 2460us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:10:19.103624 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:10:19.104075 (+ 451us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:10:19.104460 (+ 385us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:10:19.105866 (+ 1406us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:10:19.105945 (+ 79us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:10:19.105968 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 20:10:19.105986 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:10:19.106055 (+ 69us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 20:10:19.106073 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:10:19.106309 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0923 20:10:19.106410 (+ 101us) server_negotiation.cc:299] Negotiation successful
0923 20:10:19.106720 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":586,"thread_start_us":404,"threads_started":1}
I20260923 20:10:19.111209 11604 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5627a53cb9e7401695a394805cda654a: Bootstrap starting.
I20260923 20:10:19.114204 11604 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5627a53cb9e7401695a394805cda654a: Neither blocks nor log segments found. Creating new log.
I20260923 20:10:19.115280 11604 log.cc:824] T 00000000000000000000000000000000 P 5627a53cb9e7401695a394805cda654a: Log is configured to *not* fsync() on all Append() calls
I20260923 20:10:19.118363 11604 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5627a53cb9e7401695a394805cda654a: No bootstrap required, opened a new log
I20260923 20:10:19.125267 11604 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5627a53cb9e7401695a394805cda654a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5627a53cb9e7401695a394805cda654a" member_type: VOTER last_known_addr { host: "127.1.170.190" port: 45817 } }
I20260923 20:10:19.125945 11604 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5627a53cb9e7401695a394805cda654a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:10:19.126222 11604 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5627a53cb9e7401695a394805cda654a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5627a53cb9e7401695a394805cda654a, State: Initialized, Role: FOLLOWER
I20260923 20:10:19.126876 11604 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5627a53cb9e7401695a394805cda654a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5627a53cb9e7401695a394805cda654a" member_type: VOTER last_known_addr { host: "127.1.170.190" port: 45817 } }
I20260923 20:10:19.127079 11604 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5627a53cb9e7401695a394805cda654a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 20:10:19.127236 11604 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5627a53cb9e7401695a394805cda654a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 20:10:19.127420 11604 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5627a53cb9e7401695a394805cda654a [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:10:19.130470 11604 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5627a53cb9e7401695a394805cda654a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5627a53cb9e7401695a394805cda654a" member_type: VOTER last_known_addr { host: "127.1.170.190" port: 45817 } }
I20260923 20:10:19.131122 11604 leader_election.cc:302] T 00000000000000000000000000000000 P 5627a53cb9e7401695a394805cda654a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5627a53cb9e7401695a394805cda654a; no voters:
I20260923 20:10:19.131807 11604 leader_election.cc:288] T 00000000000000000000000000000000 P 5627a53cb9e7401695a394805cda654a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 20:10:19.131995 11609 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5627a53cb9e7401695a394805cda654a [term 1 FOLLOWER]: Leader election won for term 1
I20260923 20:10:19.132721 11609 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5627a53cb9e7401695a394805cda654a [term 1 LEADER]: Becoming Leader. State: Replica: 5627a53cb9e7401695a394805cda654a, State: Running, Role: LEADER
I20260923 20:10:19.133298 11609 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5627a53cb9e7401695a394805cda654a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5627a53cb9e7401695a394805cda654a" member_type: VOTER last_known_addr { host: "127.1.170.190" port: 45817 } }
I20260923 20:10:19.134035 11604 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5627a53cb9e7401695a394805cda654a [sys.catalog]: configured and running, proceeding with master startup.
I20260923 20:10:19.137006 11611 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5627a53cb9e7401695a394805cda654a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5627a53cb9e7401695a394805cda654a. Latest consensus state: current_term: 1 leader_uuid: "5627a53cb9e7401695a394805cda654a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5627a53cb9e7401695a394805cda654a" member_type: VOTER last_known_addr { host: "127.1.170.190" port: 45817 } } }
I20260923 20:10:19.137101 11610 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5627a53cb9e7401695a394805cda654a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5627a53cb9e7401695a394805cda654a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5627a53cb9e7401695a394805cda654a" member_type: VOTER last_known_addr { host: "127.1.170.190" port: 45817 } } }
I20260923 20:10:19.137471 11611 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5627a53cb9e7401695a394805cda654a [sys.catalog]: This master's current role is: LEADER
I20260923 20:10:19.137547 11610 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5627a53cb9e7401695a394805cda654a [sys.catalog]: This master's current role is: LEADER
I20260923 20:10:19.147379 11616 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 20:10:19.152552 11616 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 20:10:19.161942 11616 catalog_manager.cc:1347] Generated new cluster ID: 2d047de27c744f90b7b229b43584ec3e
I20260923 20:10:19.162077 11616 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 20:10:19.184019 11616 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 20:10:19.184597 11616 catalog_manager.cc:1504] Loading token signing keys...
I20260923 20:10:19.200937 11616 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5627a53cb9e7401695a394805cda654a: Generated new TSK 0
I20260923 20:10:19.201923 11616 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 20:10:19.210091 1706 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqUuku6/build/asan/bin/kudu
/tmp/dist-test-taskqUuku6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790194121577027-1706-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790194121577027-1706-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790194121577027-1706-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790194121577027-1706-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.170.129:0
--local_ip_for_outbound_sockets=127.1.170.129
--webserver_interface=127.1.170.129
--webserver_port=0
--tserver_master_addrs=127.1.170.190:45817
--builtin_ntp_servers=127.1.170.148:42537
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260923 20:10:19.554315 11628 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:10:19.554622 11628 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:10:19.554796 11628 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:10:19.562808 11628 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:10:19.562935 11628 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:10:19.563109 11628 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.170.129
I20260923 20:10:19.572960 11628 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.170.148:42537
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790194121577027-1706-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790194121577027-1706-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.170.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790194121577027-1706-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.170.129
--webserver_port=0
--tserver_master_addrs=127.1.170.190:45817
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.170.129
--log_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790194121577027-1706-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:10:19.574420 11628 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:10:19.576222 11628 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:10:19.589896 11633 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:10:19.589946 11636 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:10:19.590018 11634 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:10:19.591763 11628 server_base.cc:1034] running on GCE node
I20260923 20:10:19.592469 11628 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:10:19.593492 11628 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:10:19.594707 11628 hybrid_clock.cc:648] HybridClock initialized: now 1790194219594642 us; error 65 us; skew 500 ppm
I20260923 20:10:19.595060 11628 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260923 20:10:19.597491 11628 webserver.cc:460] Webserver started at http://127.1.170.129:40231/ using document root <none> and password file <none>
I20260923 20:10:19.598284 11628 fs_manager.cc:359] Metadata directory not provided
I20260923 20:10:19.598376 11628 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:10:19.598730 11628 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:10:19.601627 11628 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790194121577027-1706-0/minicluster-data/ts-0/data/instance:
uuid: "d9360b421c234590b0513b6e72545957"
format_stamp: "Formatted at 2026-09-23 20:10:19 on dist-test-slave-th4g"
server_key: "a52e8c2c9685ceb0607af3b9d98616ad"
server_key_iv: "33326c6ab6d683cae265636cb94aaff1"
server_key_version: "encryptionkey@0"
I20260923 20:10:19.603299 11628 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790194121577027-1706-0/minicluster-data/ts-0/wal/instance:
uuid: "d9360b421c234590b0513b6e72545957"
format_stamp: "Formatted at 2026-09-23 20:10:19 on dist-test-slave-th4g"
server_key: "a52e8c2c9685ceb0607af3b9d98616ad"
server_key_iv: "33326c6ab6d683cae265636cb94aaff1"
server_key_version: "encryptionkey@0"
I20260923 20:10:19.610281 11628 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.007s sys 0.000s
I20260923 20:10:19.614566 11642 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:10:19.616058 11628 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260923 20:10:19.616242 11628 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790194121577027-1706-0/minicluster-data/ts-0/data,/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790194121577027-1706-0/minicluster-data/ts-0/wal
uuid: "d9360b421c234590b0513b6e72545957"
format_stamp: "Formatted at 2026-09-23 20:10:19 on dist-test-slave-th4g"
server_key: "a52e8c2c9685ceb0607af3b9d98616ad"
server_key_iv: "33326c6ab6d683cae265636cb94aaff1"
server_key_version: "encryptionkey@0"
I20260923 20:10:19.616484 11628 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790194121577027-1706-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790194121577027-1706-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790194121577027-1706-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:10:19.622570 11628 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:10:19.623380 11628 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:10:19.623646 11628 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:10:19.624560 11628 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 20:10:19.626149 11628 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 20:10:19.626258 11628 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 20:10:19.626399 11628 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 20:10:19.626475 11628 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 20:10:19.660408 11628 rpc_server.cc:307] RPC server started. Bound to: 127.1.170.129:45067
I20260923 20:10:19.660499 11754 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.170.129:45067 every 8 connection(s)
I20260923 20:10:19.661839 11628 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790194121577027-1706-0/minicluster-data/ts-0/data/info.pb
I20260923 20:10:19.663468 1706 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqUuku6/build/asan/bin/kudu as pid 11628
I20260923 20:10:19.663646 1706 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790194121577027-1706-0/minicluster-data/ts-0/wal/instance
I20260923 20:10:19.664422 1706 external_mini_cluster.cc:1428] Setting key 8f04a606bcafe49a4a50d993f3ac3c87
I20260923 20:10:19.667428 1706 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqUuku6/build/asan/bin/kudu
/tmp/dist-test-taskqUuku6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790194121577027-1706-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790194121577027-1706-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790194121577027-1706-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790194121577027-1706-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.170.130:0
--local_ip_for_outbound_sockets=127.1.170.130
--webserver_interface=127.1.170.130
--webserver_port=0
--tserver_master_addrs=127.1.170.190:45817
--builtin_ntp_servers=127.1.170.148:42537
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260923 20:10:19.670796 11757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:10:19.664542 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.170.129:60011 (local address 127.1.170.190:45817)
0923 20:10:19.665004 (+ 462us) server_negotiation.cc:206] Beginning negotiation
0923 20:10:19.665011 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:10:19.666540 (+ 1529us) server_negotiation.cc:407] Connection header received
0923 20:10:19.668376 (+ 1836us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:10:19.668385 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:10:19.668454 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:10:19.668597 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:10:19.669986 (+ 1389us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:10:19.669994 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:10:19.670001 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 20:10:19.670023 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:10:19.670076 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 20:10:19.670093 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:10:19.670223 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0923 20:10:19.670362 (+ 139us) server_negotiation.cc:299] Negotiation successful
0923 20:10:19.670437 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":338,"thread_start_us":222,"threads_started":1}
I20260923 20:10:19.671607 11758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:10:19.665135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.190:45817 (local address 127.1.170.129:60011)
0923 20:10:19.666354 (+ 1219us) negotiation.cc:107] Waiting for socket to connect
0923 20:10:19.666409 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0923 20:10:19.668072 (+ 1663us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:10:19.668841 (+ 769us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:10:19.668849 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:10:19.668943 (+ 94us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:10:19.669664 (+ 721us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 20:10:19.669667 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:10:19.669710 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 20:10:19.669792 (+ 82us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:10:19.670184 (+ 392us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:10:19.670189 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:10:19.670195 (+ 6us) client_negotiation.cc:769] Sending connection context
0923 20:10:19.670398 (+ 203us) client_negotiation.cc:240] Negotiation successful
0923 20:10:19.670593 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":521,"thread_start_us":301,"threads_started":1}
I20260923 20:10:19.673219 11755 heartbeater.cc:347] Connected to a master server at 127.1.170.190:45817
I20260923 20:10:19.673527 11755 heartbeater.cc:464] Registering TS with master...
I20260923 20:10:19.674171 11755 heartbeater.cc:511] Master 127.1.170.190:45817 requested a full tablet report, sending...
I20260923 20:10:19.676544 11569 ts_manager.cc:194] Registered new tserver with Master: d9360b421c234590b0513b6e72545957 (127.1.170.129:45067)
W20260923 20:10:20.019147 11759 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:10:20.019486 11759 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:10:20.019658 11759 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:10:20.028090 11759 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:10:20.028270 11759 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:10:20.028487 11759 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.170.130
I20260923 20:10:20.038358 11759 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.170.148:42537
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790194121577027-1706-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790194121577027-1706-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.170.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790194121577027-1706-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.170.130
--webserver_port=0
--tserver_master_addrs=127.1.170.190:45817
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.170.130
--log_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790194121577027-1706-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:10:20.039690 11759 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:10:20.041605 11759 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:10:20.055001 11765 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:10:20.054980 11767 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:10:20.055218 11764 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:10:20.055985 11759 server_base.cc:1034] running on GCE node
I20260923 20:10:20.056612 11759 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:10:20.057667 11759 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:10:20.058915 11759 hybrid_clock.cc:648] HybridClock initialized: now 1790194220058854 us; error 53 us; skew 500 ppm
I20260923 20:10:20.059247 11759 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260923 20:10:20.061939 11759 webserver.cc:460] Webserver started at http://127.1.170.130:37069/ using document root <none> and password file <none>
I20260923 20:10:20.062700 11759 fs_manager.cc:359] Metadata directory not provided
I20260923 20:10:20.062836 11759 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:10:20.063165 11759 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:10:20.066274 11759 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790194121577027-1706-0/minicluster-data/ts-1/data/instance:
uuid: "a094680b848d46438c4b022b25d9edf6"
format_stamp: "Formatted at 2026-09-23 20:10:20 on dist-test-slave-th4g"
server_key: "e278c610cdf2ea803c0846ae2bb77d7f"
server_key_iv: "afc9bca42865e3a5e897bd330a2c0a41"
server_key_version: "encryptionkey@0"
I20260923 20:10:20.067157 11759 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790194121577027-1706-0/minicluster-data/ts-1/wal/instance:
uuid: "a094680b848d46438c4b022b25d9edf6"
format_stamp: "Formatted at 2026-09-23 20:10:20 on dist-test-slave-th4g"
server_key: "e278c610cdf2ea803c0846ae2bb77d7f"
server_key_iv: "afc9bca42865e3a5e897bd330a2c0a41"
server_key_version: "encryptionkey@0"
I20260923 20:10:20.073362 11759 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.001s
I20260923 20:10:20.078379 11773 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:10:20.080222 11759 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260923 20:10:20.080461 11759 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790194121577027-1706-0/minicluster-data/ts-1/data,/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790194121577027-1706-0/minicluster-data/ts-1/wal
uuid: "a094680b848d46438c4b022b25d9edf6"
format_stamp: "Formatted at 2026-09-23 20:10:20 on dist-test-slave-th4g"
server_key: "e278c610cdf2ea803c0846ae2bb77d7f"
server_key_iv: "afc9bca42865e3a5e897bd330a2c0a41"
server_key_version: "encryptionkey@0"
I20260923 20:10:20.080731 11759 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790194121577027-1706-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790194121577027-1706-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790194121577027-1706-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:10:20.086737 11759 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:10:20.087587 11759 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:10:20.087872 11759 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:10:20.088850 11759 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 20:10:20.090356 11759 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 20:10:20.090477 11759 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 20:10:20.090608 11759 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 20:10:20.090695 11759 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 20:10:20.127517 11759 rpc_server.cc:307] RPC server started. Bound to: 127.1.170.130:33035
I20260923 20:10:20.127779 11885 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.170.130:33035 every 8 connection(s)
I20260923 20:10:20.128899 11759 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790194121577027-1706-0/minicluster-data/ts-1/data/info.pb
I20260923 20:10:20.132781 1706 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqUuku6/build/asan/bin/kudu as pid 11759
I20260923 20:10:20.132946 1706 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790194121577027-1706-0/minicluster-data/ts-1/wal/instance
I20260923 20:10:20.133455 1706 external_mini_cluster.cc:1428] Setting key c852ec3ae7d8c0aa16226c84019d5755
I20260923 20:10:20.136391 1706 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqUuku6/build/asan/bin/kudu
/tmp/dist-test-taskqUuku6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790194121577027-1706-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790194121577027-1706-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790194121577027-1706-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790194121577027-1706-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.170.131:0
--local_ip_for_outbound_sockets=127.1.170.131
--webserver_interface=127.1.170.131
--webserver_port=0
--tserver_master_addrs=127.1.170.190:45817
--builtin_ntp_servers=127.1.170.148:42537
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260923 20:10:20.139293 11757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:10:20.132121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.170.130:49441 (local address 127.1.170.190:45817)
0923 20:10:20.132376 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0923 20:10:20.132380 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 20:10:20.134260 (+ 1880us) server_negotiation.cc:407] Connection header received
0923 20:10:20.136171 (+ 1911us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:10:20.136178 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:10:20.136330 (+ 152us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:10:20.136455 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:10:20.138142 (+ 1687us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:10:20.138153 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:10:20.138161 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 20:10:20.138192 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:10:20.138252 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 20:10:20.138278 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:10:20.138481 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0923 20:10:20.138748 (+ 267us) server_negotiation.cc:299] Negotiation successful
0923 20:10:20.138834 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
I20260923 20:10:20.140115 11888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:10:20.132756 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.190:45817 (local address 127.1.170.130:49441)
0923 20:10:20.134057 (+ 1301us) negotiation.cc:107] Waiting for socket to connect
0923 20:10:20.134104 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0923 20:10:20.135894 (+ 1790us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:10:20.136764 (+ 870us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:10:20.136773 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:10:20.136870 (+ 97us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:10:20.137679 (+ 809us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 20:10:20.137682 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:10:20.137731 (+ 49us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 20:10:20.137828 (+ 97us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:10:20.138469 (+ 641us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:10:20.138478 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:10:20.138505 (+ 27us) client_negotiation.cc:769] Sending connection context
0923 20:10:20.138777 (+ 272us) client_negotiation.cc:240] Negotiation successful
0923 20:10:20.139005 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":667,"thread_start_us":446,"threads_started":1}
I20260923 20:10:20.141691 11886 heartbeater.cc:347] Connected to a master server at 127.1.170.190:45817
I20260923 20:10:20.142087 11886 heartbeater.cc:464] Registering TS with master...
I20260923 20:10:20.142709 11886 heartbeater.cc:511] Master 127.1.170.190:45817 requested a full tablet report, sending...
I20260923 20:10:20.144260 11569 ts_manager.cc:194] Registered new tserver with Master: a094680b848d46438c4b022b25d9edf6 (127.1.170.130:33035)
W20260923 20:10:20.475262 11889 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:10:20.475592 11889 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:10:20.475684 11889 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:10:20.483215 11889 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:10:20.483347 11889 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:10:20.483460 11889 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.170.131
I20260923 20:10:20.493165 11889 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.170.148:42537
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790194121577027-1706-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790194121577027-1706-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.170.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790194121577027-1706-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.170.131
--webserver_port=0
--tserver_master_addrs=127.1.170.190:45817
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.170.131
--log_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790194121577027-1706-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:10:20.494513 11889 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:10:20.496243 11889 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:10:20.508924 11895 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:10:20.509160 11894 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:10:20.510478 11897 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:10:20.511348 11889 server_base.cc:1034] running on GCE node
I20260923 20:10:20.512182 11889 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:10:20.514070 11889 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:10:20.515367 11889 hybrid_clock.cc:648] HybridClock initialized: now 1790194220515298 us; error 65 us; skew 500 ppm
I20260923 20:10:20.515760 11889 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260923 20:10:20.518391 11889 webserver.cc:460] Webserver started at http://127.1.170.131:39441/ using document root <none> and password file <none>
I20260923 20:10:20.519151 11889 fs_manager.cc:359] Metadata directory not provided
I20260923 20:10:20.519272 11889 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:10:20.519632 11889 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:10:20.522747 11889 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790194121577027-1706-0/minicluster-data/ts-2/data/instance:
uuid: "0414baaf2fd445f9bf840b65821c1b60"
format_stamp: "Formatted at 2026-09-23 20:10:20 on dist-test-slave-th4g"
server_key: "42b91761b8a0ec834c1d3d266b3d9465"
server_key_iv: "ea5213f4fc31393ba1357546b859221d"
server_key_version: "encryptionkey@0"
I20260923 20:10:20.523563 11889 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790194121577027-1706-0/minicluster-data/ts-2/wal/instance:
uuid: "0414baaf2fd445f9bf840b65821c1b60"
format_stamp: "Formatted at 2026-09-23 20:10:20 on dist-test-slave-th4g"
server_key: "42b91761b8a0ec834c1d3d266b3d9465"
server_key_iv: "ea5213f4fc31393ba1357546b859221d"
server_key_version: "encryptionkey@0"
I20260923 20:10:20.529395 11889 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260923 20:10:20.535367 11903 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:10:20.537560 11889 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260923 20:10:20.537735 11889 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790194121577027-1706-0/minicluster-data/ts-2/data,/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790194121577027-1706-0/minicluster-data/ts-2/wal
uuid: "0414baaf2fd445f9bf840b65821c1b60"
format_stamp: "Formatted at 2026-09-23 20:10:20 on dist-test-slave-th4g"
server_key: "42b91761b8a0ec834c1d3d266b3d9465"
server_key_iv: "ea5213f4fc31393ba1357546b859221d"
server_key_version: "encryptionkey@0"
I20260923 20:10:20.538049 11889 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790194121577027-1706-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790194121577027-1706-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790194121577027-1706-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:10:20.544812 11889 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:10:20.545686 11889 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:10:20.546051 11889 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:10:20.547066 11889 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 20:10:20.548790 11889 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 20:10:20.548885 11889 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 20:10:20.549036 11889 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 20:10:20.549136 11889 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 20:10:20.587409 11889 rpc_server.cc:307] RPC server started. Bound to: 127.1.170.131:38863
I20260923 20:10:20.587502 12015 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.170.131:38863 every 8 connection(s)
I20260923 20:10:20.588811 11889 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790194121577027-1706-0/minicluster-data/ts-2/data/info.pb
I20260923 20:10:20.592878 1706 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqUuku6/build/asan/bin/kudu as pid 11889
I20260923 20:10:20.593045 1706 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790194121577027-1706-0/minicluster-data/ts-2/wal/instance
I20260923 20:10:20.593508 1706 external_mini_cluster.cc:1428] Setting key 68933d4b928ac6a96637170c4117be4f
I20260923 20:10:20.599349 11757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:10:20.592520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.170.131:48017 (local address 127.1.170.190:45817)
0923 20:10:20.592759 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0923 20:10:20.592766 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:10:20.594508 (+ 1742us) server_negotiation.cc:407] Connection header received
0923 20:10:20.596507 (+ 1999us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:10:20.596513 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:10:20.596585 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:10:20.596690 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:10:20.598650 (+ 1960us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:10:20.598655 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:10:20.598659 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 20:10:20.598674 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:10:20.598705 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 20:10:20.598759 (+ 54us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:10:20.598867 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0923 20:10:20.599060 (+ 193us) server_negotiation.cc:299] Negotiation successful
0923 20:10:20.599111 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260923 20:10:20.600435 12018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:10:20.593158 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.190:45817 (local address 127.1.170.131:48017)
0923 20:10:20.594348 (+ 1190us) negotiation.cc:107] Waiting for socket to connect
0923 20:10:20.594393 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0923 20:10:20.596150 (+ 1757us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:10:20.597191 (+ 1041us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:10:20.597205 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:10:20.597358 (+ 153us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:10:20.598308 (+ 950us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 20:10:20.598311 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:10:20.598355 (+ 44us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 20:10:20.598462 (+ 107us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:10:20.598862 (+ 400us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:10:20.598870 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:10:20.598876 (+ 6us) client_negotiation.cc:769] Sending connection context
0923 20:10:20.599120 (+ 244us) client_negotiation.cc:240] Negotiation successful
0923 20:10:20.599386 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":585,"thread_start_us":373,"threads_started":1}
I20260923 20:10:20.602025 12016 heartbeater.cc:347] Connected to a master server at 127.1.170.190:45817
I20260923 20:10:20.602397 12016 heartbeater.cc:464] Registering TS with master...
I20260923 20:10:20.603039 12016 heartbeater.cc:511] Master 127.1.170.190:45817 requested a full tablet report, sending...
I20260923 20:10:20.604573 11569 ts_manager.cc:194] Registered new tserver with Master: 0414baaf2fd445f9bf840b65821c1b60 (127.1.170.131:38863)
I20260923 20:10:20.609695 1706 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260923 20:10:20.615919 11757 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 20:10:20.614633 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50792 (local address 127.1.170.190:45817)
0923 20:10:20.614917 (+ 284us) server_negotiation.cc:206] Beginning negotiation
0923 20:10:20.614923 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:10:20.615088 (+ 165us) server_negotiation.cc:407] Connection header received
0923 20:10:20.615182 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:10:20.615186 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:10:20.615244 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:10:20.615337 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:10:20.615688 (+ 351us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50792: BlockingRecv error: recv got EOF from 127.0.0.1:50792 (error 108)
Metrics: {"server-negotiator.queue_time_us":106}
I20260923 20:10:20.616376 1706 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskqUuku6/build/asan/bin/kudu pid 11628
I20260923 20:10:20.620338 12025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:10:20.617391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34990 (local address 127.1.170.129:45067)
0923 20:10:20.618046 (+ 655us) server_negotiation.cc:206] Beginning negotiation
0923 20:10:20.618054 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 20:10:20.618083 (+ 29us) server_negotiation.cc:407] Connection header received
0923 20:10:20.618153 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:10:20.618159 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:10:20.618394 (+ 235us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:10:20.618532 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:10:20.619536 (+ 1004us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:10:20.619544 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:10:20.619556 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 20:10:20.619580 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:10:20.619627 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 20:10:20.619642 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:10:20.619763 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0923 20:10:20.619916 (+ 153us) server_negotiation.cc:299] Negotiation successful
0923 20:10:20.620053 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":467,"thread_start_us":278,"threads_started":1}
I20260923 20:10:20.621723 11750 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:10:20.703639 11755 heartbeater.cc:503] Master 127.1.170.190:45817 was elected leader, sending a full tablet report...
I20260923 20:10:20.706166 1706 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqUuku6/build/asan/bin/kudu with pid 11628
I20260923 20:10:20.725430 1706 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskqUuku6/build/asan/bin/kudu pid 11759
I20260923 20:10:20.730126 12027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:10:20.726707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34146 (local address 127.1.170.130:33035)
0923 20:10:20.727246 (+ 539us) server_negotiation.cc:206] Beginning negotiation
0923 20:10:20.727253 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:10:20.727289 (+ 36us) server_negotiation.cc:407] Connection header received
0923 20:10:20.727359 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:10:20.727366 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:10:20.727593 (+ 227us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:10:20.727719 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:10:20.729038 (+ 1319us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:10:20.729050 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:10:20.729067 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 20:10:20.729102 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:10:20.729171 (+ 69us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 20:10:20.729193 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:10:20.729352 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0923 20:10:20.729526 (+ 174us) server_negotiation.cc:299] Negotiation successful
0923 20:10:20.729707 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":413,"thread_start_us":296,"threads_started":1}
I20260923 20:10:20.731941 11881 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:10:20.821211 1706 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqUuku6/build/asan/bin/kudu with pid 11759
I20260923 20:10:20.837651 1706 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskqUuku6/build/asan/bin/kudu pid 11889
I20260923 20:10:20.842347 12029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:10:20.838656 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37818 (local address 127.1.170.131:38863)
0923 20:10:20.839370 (+ 714us) server_negotiation.cc:206] Beginning negotiation
0923 20:10:20.839377 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:10:20.839409 (+ 32us) server_negotiation.cc:407] Connection header received
0923 20:10:20.839479 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:10:20.839486 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:10:20.839716 (+ 230us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:10:20.839919 (+ 203us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:10:20.841070 (+ 1151us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:10:20.841076 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:10:20.841088 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 20:10:20.841103 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:10:20.841140 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 20:10:20.841153 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:10:20.841359 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0923 20:10:20.841429 (+ 70us) server_negotiation.cc:299] Negotiation successful
0923 20:10:20.841706 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":536,"thread_start_us":377,"threads_started":1}
I20260923 20:10:20.843755 12011 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:10:20.926319 1706 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqUuku6/build/asan/bin/kudu with pid 11889
I20260923 20:10:20.943504 1706 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskqUuku6/build/asan/bin/kudu pid 11538
I20260923 20:10:20.944994 11599 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:10:21.046559 1706 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqUuku6/build/asan/bin/kudu with pid 11538
2026-09-23T20:10:21Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (2442 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790194121577027-1706-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 20:10:21 dist-test-slave-th4g krb5kdc[12034](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 20:10:21 dist-test-slave-th4g krb5kdc[12034](info): set up 2 sockets
Sep 23 20:10:21 dist-test-slave-th4g krb5kdc[12034](info): commencing operation
krb5kdc: starting...
W20260923 20:10:22.155128 1706 mini_kdc.cc:121] Time spent starting KDC: real 1.067s user 0.000s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790194121577027-1706-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790194121577027-1706-0/krb5kdc/test-user.keytab.
Sep 23 20:10:22 dist-test-slave-th4g krb5kdc[12034](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194222, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T20:10:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T20:10:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.170.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.170.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.170.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790194121577027-1706-0/krb5kdc/kudu_127.1.170.190.keytab.
Entry for principal kudu/127.1.170.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790194121577027-1706-0/krb5kdc/kudu_127.1.170.190.keytab.
I20260923 20:10:22.390401 1706 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqUuku6/build/asan/bin/kudu
/tmp/dist-test-taskqUuku6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790194121577027-1706-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790194121577027-1706-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790194121577027-1706-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790194121577027-1706-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.170.190:34019
--webserver_interface=127.1.170.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790194121577027-1706-0/krb5kdc/kudu_127.1.170.190.keytab
--principal=kudu/127.1.170.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.170.148:34487
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.170.190:34019
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790194121577027-1706-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790194121577027-1706-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790194121577027-1706-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:10:22.717326 12047 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:10:22.717618 12047 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:10:22.717772 12047 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:10:22.725142 12047 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 20:10:22.725251 12047 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:10:22.725342 12047 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:10:22.725440 12047 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 20:10:22.725528 12047 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 20:10:22.735468 12047 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.170.148:34487
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790194121577027-1706-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790194121577027-1706-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.170.190:34019
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790194121577027-1706-0/krb5kdc/kudu_127.1.170.190.keytab
--principal=kudu/127.1.170.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.170.190:34019
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790194121577027-1706-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.170.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790194121577027-1706-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:10:22.736804 12047 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:10:22.738155 12047 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:10:22.750308 12053 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:10:22.750253 12052 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:10:22.750280 12055 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:10:22.751026 12047 server_base.cc:1034] running on GCE node
I20260923 20:10:22.751819 12047 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:10:22.753237 12047 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:10:22.754470 12047 hybrid_clock.cc:648] HybridClock initialized: now 1790194222754424 us; error 52 us; skew 500 ppm
I20260923 20:10:22.754830 12047 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:10:22 dist-test-slave-th4g krb5kdc[12034](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194222, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.170.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:10:22.758200 12047 init.cc:377] Logged in from keytab as kudu/127.1.170.190@KRBTEST.COM (short username kudu)
I20260923 20:10:22.759658 12047 webserver.cc:460] Webserver started at http://127.1.170.190:40517/ using document root <none> and password file <none>
I20260923 20:10:22.760365 12047 fs_manager.cc:359] Metadata directory not provided
I20260923 20:10:22.760516 12047 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:10:22.760833 12047 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:10:22.763906 12047 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790194121577027-1706-0/minicluster-data/master-0/data/instance:
uuid: "0e40b98f6ac647018944146aecfb233a"
format_stamp: "Formatted at 2026-09-23 20:10:22 on dist-test-slave-th4g"
server_key: "7afe5774be479b48b43f1a0d6c3288ef"
server_key_iv: "14ade28af84769b485795c45b3cd01dc"
server_key_version: "encryptionkey@0"
I20260923 20:10:22.764688 12047 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790194121577027-1706-0/minicluster-data/master-0/wal/instance:
uuid: "0e40b98f6ac647018944146aecfb233a"
format_stamp: "Formatted at 2026-09-23 20:10:22 on dist-test-slave-th4g"
server_key: "7afe5774be479b48b43f1a0d6c3288ef"
server_key_iv: "14ade28af84769b485795c45b3cd01dc"
server_key_version: "encryptionkey@0"
I20260923 20:10:22.770287 12047 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260923 20:10:22.773710 12062 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:10:22.775039 12047 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 20:10:22.775218 12047 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790194121577027-1706-0/minicluster-data/master-0/data,/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790194121577027-1706-0/minicluster-data/master-0/wal
uuid: "0e40b98f6ac647018944146aecfb233a"
format_stamp: "Formatted at 2026-09-23 20:10:22 on dist-test-slave-th4g"
server_key: "7afe5774be479b48b43f1a0d6c3288ef"
server_key_iv: "14ade28af84769b485795c45b3cd01dc"
server_key_version: "encryptionkey@0"
I20260923 20:10:22.775414 12047 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790194121577027-1706-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790194121577027-1706-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790194121577027-1706-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:10:22.798025 12047 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:10:22.802451 12047 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:10:22.802760 12047 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:10:22.821094 12047 rpc_server.cc:307] RPC server started. Bound to: 127.1.170.190:34019
I20260923 20:10:22.821131 12113 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.170.190:34019 every 8 connection(s)
I20260923 20:10:22.822425 12047 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790194121577027-1706-0/minicluster-data/master-0/data/info.pb
I20260923 20:10:22.823587 1706 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqUuku6/build/asan/bin/kudu as pid 12047
I20260923 20:10:22.823796 1706 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790194121577027-1706-0/minicluster-data/master-0/wal/instance
I20260923 20:10:22.824564 1706 external_mini_cluster.cc:1428] Setting key 50d47d5e946db1629e1530274618a2c5
I20260923 20:10:22.827198 12114 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 23 20:10:22 dist-test-slave-th4g krb5kdc[12034](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194222, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.170.190@KRBTEST.COM
I20260923 20:10:22.840754 12114 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0e40b98f6ac647018944146aecfb233a: Bootstrap starting.
I20260923 20:10:22.844573 12114 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0e40b98f6ac647018944146aecfb233a: Neither blocks nor log segments found. Creating new log.
I20260923 20:10:22.845794 12114 log.cc:824] T 00000000000000000000000000000000 P 0e40b98f6ac647018944146aecfb233a: Log is configured to *not* fsync() on all Append() calls
I20260923 20:10:22.848157 12117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:10:22.826591 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51944 (local address 127.1.170.190:34019)
0923 20:10:22.827372 (+ 781us) server_negotiation.cc:206] Beginning negotiation
0923 20:10:22.827383 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0923 20:10:22.827422 (+ 39us) server_negotiation.cc:407] Connection header received
0923 20:10:22.829008 (+ 1586us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:10:22.829029 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:10:22.829438 (+ 409us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:10:22.829792 (+ 354us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:10:22.831588 (+ 1796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:10:22.832941 (+ 1353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:10:22.834821 (+ 1880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:10:22.835471 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:10:22.835598 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:10:22.840405 (+ 4807us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:10:22.840413 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:10:22.840429 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:10:22.840467 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:10:22.843681 (+ 3214us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:10:22.844607 (+ 926us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:10:22.844614 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:10:22.844620 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:10:22.844866 (+ 246us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:10:22.845327 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:10:22.845330 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:10:22.845333 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:10:22.845936 (+ 603us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:10:22.846284 (+ 348us) server_negotiation.cc:1035] Waiting for connection context
0923 20:10:22.846723 (+ 439us) server_negotiation.cc:299] Negotiation successful
0923 20:10:22.847152 (+ 429us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":552,"thread_start_us":356,"threads_started":1}
I20260923 20:10:22.848934 12114 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0e40b98f6ac647018944146aecfb233a: No bootstrap required, opened a new log
I20260923 20:10:22.856388 12114 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0e40b98f6ac647018944146aecfb233a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e40b98f6ac647018944146aecfb233a" member_type: VOTER last_known_addr { host: "127.1.170.190" port: 34019 } }
I20260923 20:10:22.856812 12114 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0e40b98f6ac647018944146aecfb233a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:10:22.856958 12114 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0e40b98f6ac647018944146aecfb233a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e40b98f6ac647018944146aecfb233a, State: Initialized, Role: FOLLOWER
I20260923 20:10:22.857584 12114 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0e40b98f6ac647018944146aecfb233a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e40b98f6ac647018944146aecfb233a" member_type: VOTER last_known_addr { host: "127.1.170.190" port: 34019 } }
I20260923 20:10:22.857832 12114 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0e40b98f6ac647018944146aecfb233a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 20:10:22.857998 12114 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0e40b98f6ac647018944146aecfb233a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 20:10:22.858196 12114 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0e40b98f6ac647018944146aecfb233a [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:10:22.860347 12114 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0e40b98f6ac647018944146aecfb233a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e40b98f6ac647018944146aecfb233a" member_type: VOTER last_known_addr { host: "127.1.170.190" port: 34019 } }
I20260923 20:10:22.860858 12114 leader_election.cc:302] T 00000000000000000000000000000000 P 0e40b98f6ac647018944146aecfb233a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0e40b98f6ac647018944146aecfb233a; no voters:
I20260923 20:10:22.861451 12114 leader_election.cc:288] T 00000000000000000000000000000000 P 0e40b98f6ac647018944146aecfb233a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 20:10:22.861593 12119 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0e40b98f6ac647018944146aecfb233a [term 1 FOLLOWER]: Leader election won for term 1
I20260923 20:10:22.862753 12119 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0e40b98f6ac647018944146aecfb233a [term 1 LEADER]: Becoming Leader. State: Replica: 0e40b98f6ac647018944146aecfb233a, State: Running, Role: LEADER
I20260923 20:10:22.863339 12119 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0e40b98f6ac647018944146aecfb233a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e40b98f6ac647018944146aecfb233a" member_type: VOTER last_known_addr { host: "127.1.170.190" port: 34019 } }
I20260923 20:10:22.863651 12114 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0e40b98f6ac647018944146aecfb233a [sys.catalog]: configured and running, proceeding with master startup.
I20260923 20:10:22.866466 12121 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0e40b98f6ac647018944146aecfb233a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0e40b98f6ac647018944146aecfb233a. Latest consensus state: current_term: 1 leader_uuid: "0e40b98f6ac647018944146aecfb233a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e40b98f6ac647018944146aecfb233a" member_type: VOTER last_known_addr { host: "127.1.170.190" port: 34019 } } }
I20260923 20:10:22.866458 12120 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0e40b98f6ac647018944146aecfb233a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0e40b98f6ac647018944146aecfb233a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e40b98f6ac647018944146aecfb233a" member_type: VOTER last_known_addr { host: "127.1.170.190" port: 34019 } } }
I20260923 20:10:22.866923 12120 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0e40b98f6ac647018944146aecfb233a [sys.catalog]: This master's current role is: LEADER
I20260923 20:10:22.866920 12121 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0e40b98f6ac647018944146aecfb233a [sys.catalog]: This master's current role is: LEADER
I20260923 20:10:22.871126 12125 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 20:10:22.875979 12125 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 20:10:22.889154 12125 catalog_manager.cc:1347] Generated new cluster ID: ddb7a092e6214720a36ab25a3a11ad6f
I20260923 20:10:22.889287 12125 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 20:10:22.908790 12125 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 20:10:22.910180 12125 catalog_manager.cc:1504] Loading token signing keys...
I20260923 20:10:22.917743 12125 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0e40b98f6ac647018944146aecfb233a: Generated new TSK 0
I20260923 20:10:22.918584 12125 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.170.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.170.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.170.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790194121577027-1706-0/krb5kdc/kudu_127.1.170.129.keytab.
Entry for principal kudu/127.1.170.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790194121577027-1706-0/krb5kdc/kudu_127.1.170.129.keytab.
I20260923 20:10:22.981114 1706 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqUuku6/build/asan/bin/kudu
/tmp/dist-test-taskqUuku6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790194121577027-1706-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790194121577027-1706-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790194121577027-1706-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790194121577027-1706-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.170.129:0
--local_ip_for_outbound_sockets=127.1.170.129
--webserver_interface=127.1.170.129
--webserver_port=0
--tserver_master_addrs=127.1.170.190:34019
--keytab_file=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790194121577027-1706-0/krb5kdc/kudu_127.1.170.129.keytab
--principal=kudu/127.1.170.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.170.148:34487
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790194121577027-1706-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790194121577027-1706-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790194121577027-1706-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:10:23.308797 12140 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:10:23.309091 12140 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:10:23.309232 12140 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:10:23.316762 12140 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:10:23.316896 12140 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:10:23.317080 12140 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.170.129
I20260923 20:10:23.327366 12140 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.170.148:34487
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790194121577027-1706-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790194121577027-1706-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790194121577027-1706-0/krb5kdc/kudu_127.1.170.129.keytab
--principal=kudu/127.1.170.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.170.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790194121577027-1706-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.170.129
--webserver_port=0
--tserver_master_addrs=127.1.170.190:34019
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.170.129
--log_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790194121577027-1706-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:10:23.328723 12140 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:10:23.330425 12140 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:10:23.344707 12146 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:10:23.344715 12148 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:10:23.345170 12145 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:10:23.346052 12140 server_base.cc:1034] running on GCE node
I20260923 20:10:23.346941 12140 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:10:23.348642 12140 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:10:23.350009 12140 hybrid_clock.cc:648] HybridClock initialized: now 1790194223349936 us; error 71 us; skew 500 ppm
I20260923 20:10:23.350375 12140 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:10:23 dist-test-slave-th4g krb5kdc[12034](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194223, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.170.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:10:23.362466 12140 init.cc:377] Logged in from keytab as kudu/127.1.170.129@KRBTEST.COM (short username kudu)
I20260923 20:10:23.364269 12140 webserver.cc:460] Webserver started at http://127.1.170.129:38351/ using document root <none> and password file <none>
I20260923 20:10:23.365032 12140 fs_manager.cc:359] Metadata directory not provided
I20260923 20:10:23.365207 12140 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:10:23.365540 12140 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:10:23.368647 12140 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790194121577027-1706-0/minicluster-data/ts-0/data/instance:
uuid: "1776720c83cc42a38935643fb249c406"
format_stamp: "Formatted at 2026-09-23 20:10:23 on dist-test-slave-th4g"
server_key: "a155e84c8f272c8029ef98f70e69c36f"
server_key_iv: "59b13cba92757d5abc05d61a42672249"
server_key_version: "encryptionkey@0"
I20260923 20:10:23.369489 12140 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790194121577027-1706-0/minicluster-data/ts-0/wal/instance:
uuid: "1776720c83cc42a38935643fb249c406"
format_stamp: "Formatted at 2026-09-23 20:10:23 on dist-test-slave-th4g"
server_key: "a155e84c8f272c8029ef98f70e69c36f"
server_key_iv: "59b13cba92757d5abc05d61a42672249"
server_key_version: "encryptionkey@0"
I20260923 20:10:23.375298 12140 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260923 20:10:23.379125 12155 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:10:23.381191 12140 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260923 20:10:23.381402 12140 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790194121577027-1706-0/minicluster-data/ts-0/data,/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790194121577027-1706-0/minicluster-data/ts-0/wal
uuid: "1776720c83cc42a38935643fb249c406"
format_stamp: "Formatted at 2026-09-23 20:10:23 on dist-test-slave-th4g"
server_key: "a155e84c8f272c8029ef98f70e69c36f"
server_key_iv: "59b13cba92757d5abc05d61a42672249"
server_key_version: "encryptionkey@0"
I20260923 20:10:23.381623 12140 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790194121577027-1706-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790194121577027-1706-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790194121577027-1706-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:10:23.400063 12140 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:10:23.405071 12140 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:10:23.405445 12140 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:10:23.406517 12140 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 20:10:23.407945 12140 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 20:10:23.408021 12140 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 20:10:23.408164 12140 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 20:10:23.408222 12140 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 20:10:23.444623 12140 rpc_server.cc:307] RPC server started. Bound to: 127.1.170.129:39829
I20260923 20:10:23.444669 12267 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.170.129:39829 every 8 connection(s)
I20260923 20:10:23.445853 12140 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790194121577027-1706-0/minicluster-data/ts-0/data/info.pb
I20260923 20:10:23.454329 1706 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqUuku6/build/asan/bin/kudu as pid 12140
I20260923 20:10:23.454532 1706 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790194121577027-1706-0/minicluster-data/ts-0/wal/instance
I20260923 20:10:23.455158 1706 external_mini_cluster.cc:1428] Setting key 8b7fc266a50d06aa03c5b2dd2443e945
Sep 23 20:10:23 dist-test-slave-th4g krb5kdc[12034](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194223, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.170.129@KRBTEST.COM for kudu/127.1.170.190@KRBTEST.COM
I20260923 20:10:23.467921 12270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:10:23.448497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.170.129:56275 (local address 127.1.170.190:34019)
0923 20:10:23.448965 (+ 468us) server_negotiation.cc:206] Beginning negotiation
0923 20:10:23.448970 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 20:10:23.449775 (+ 805us) server_negotiation.cc:407] Connection header received
0923 20:10:23.451578 (+ 1803us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:10:23.451584 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:10:23.451678 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:10:23.451796 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:10:23.453289 (+ 1493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:10:23.454111 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:10:23.455988 (+ 1877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:10:23.456492 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:10:23.456604 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:10:23.460449 (+ 3845us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:10:23.460455 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:10:23.460458 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:10:23.460492 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:10:23.463575 (+ 3083us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:10:23.464931 (+ 1356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:10:23.464945 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:10:23.464948 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:10:23.465108 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:10:23.465678 (+ 570us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:10:23.465683 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:10:23.465686 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:10:23.466244 (+ 558us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:10:23.466448 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0923 20:10:23.467300 (+ 852us) server_negotiation.cc:299] Negotiation successful
0923 20:10:23.467543 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":373,"thread_start_us":270,"threads_started":1}
I20260923 20:10:23.468954 12271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:10:23.448877 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.190:34019 (local address 127.1.170.129:56275)
0923 20:10:23.449598 (+ 721us) negotiation.cc:107] Waiting for socket to connect
0923 20:10:23.449643 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0923 20:10:23.451310 (+ 1667us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:10:23.452025 (+ 715us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:10:23.452033 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:10:23.452563 (+ 530us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:10:23.453112 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:10:23.453128 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:10:23.454287 (+ 1159us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:10:23.454298 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:10:23.455765 (+ 1467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:10:23.455780 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:10:23.456786 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:10:23.456794 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:10:23.457116 (+ 322us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:10:23.457864 (+ 748us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:10:23.457974 (+ 110us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:10:23.460193 (+ 2219us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:10:23.463798 (+ 3605us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:10:23.463808 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:10:23.463849 (+ 41us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:10:23.464657 (+ 808us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:10:23.465250 (+ 593us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:10:23.465255 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:10:23.465260 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:10:23.465544 (+ 284us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:10:23.466440 (+ 896us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:10:23.466450 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:10:23.466887 (+ 437us) client_negotiation.cc:769] Sending connection context
0923 20:10:23.467193 (+ 306us) client_negotiation.cc:240] Negotiation successful
0923 20:10:23.467800 (+ 607us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":492,"thread_start_us":304,"threads_started":1}
I20260923 20:10:23.470925 12268 heartbeater.cc:347] Connected to a master server at 127.1.170.190:34019
I20260923 20:10:23.471263 12268 heartbeater.cc:464] Registering TS with master...
I20260923 20:10:23.472051 12268 heartbeater.cc:511] Master 127.1.170.190:34019 requested a full tablet report, sending...
I20260923 20:10:23.474567 12079 ts_manager.cc:194] Registered new tserver with Master: 1776720c83cc42a38935643fb249c406 (127.1.170.129:39829)
I20260923 20:10:23.476858 12079 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.170.129@KRBTEST.COM'} at 127.1.170.129:56275
WARNING: no policy specified for kudu/127.1.170.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.170.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.170.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790194121577027-1706-0/krb5kdc/kudu_127.1.170.130.keytab.
Entry for principal kudu/127.1.170.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790194121577027-1706-0/krb5kdc/kudu_127.1.170.130.keytab.
I20260923 20:10:23.514210 1706 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqUuku6/build/asan/bin/kudu
/tmp/dist-test-taskqUuku6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790194121577027-1706-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790194121577027-1706-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790194121577027-1706-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790194121577027-1706-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.170.130:0
--local_ip_for_outbound_sockets=127.1.170.130
--webserver_interface=127.1.170.130
--webserver_port=0
--tserver_master_addrs=127.1.170.190:34019
--keytab_file=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790194121577027-1706-0/krb5kdc/kudu_127.1.170.130.keytab
--principal=kudu/127.1.170.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.170.148:34487
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790194121577027-1706-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790194121577027-1706-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790194121577027-1706-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:10:23.850011 12274 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:10:23.850353 12274 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:10:23.850529 12274 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:10:23.857823 12274 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:10:23.858017 12274 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:10:23.858139 12274 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.170.130
I20260923 20:10:23.868137 12274 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.170.148:34487
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790194121577027-1706-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790194121577027-1706-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790194121577027-1706-0/krb5kdc/kudu_127.1.170.130.keytab
--principal=kudu/127.1.170.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.170.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790194121577027-1706-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.170.130
--webserver_port=0
--tserver_master_addrs=127.1.170.190:34019
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.170.130
--log_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790194121577027-1706-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:10:23.869493 12274 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:10:23.870805 12274 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:10:23.885026 12282 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:10:23.885236 12279 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:10:23.885149 12280 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:10:23.886476 12274 server_base.cc:1034] running on GCE node
I20260923 20:10:23.887220 12274 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:10:23.888373 12274 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:10:23.889595 12274 hybrid_clock.cc:648] HybridClock initialized: now 1790194223889544 us; error 37 us; skew 500 ppm
I20260923 20:10:23.890050 12274 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:10:23 dist-test-slave-th4g krb5kdc[12034](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194223, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.170.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:10:23.893397 12274 init.cc:377] Logged in from keytab as kudu/127.1.170.130@KRBTEST.COM (short username kudu)
I20260923 20:10:23.895052 12274 webserver.cc:460] Webserver started at http://127.1.170.130:35151/ using document root <none> and password file <none>
I20260923 20:10:23.895777 12274 fs_manager.cc:359] Metadata directory not provided
I20260923 20:10:23.895926 12274 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:10:23.896236 12274 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:10:23.899448 12274 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790194121577027-1706-0/minicluster-data/ts-1/data/instance:
uuid: "8579308d6fd543d4bb9b3470d794dbf5"
format_stamp: "Formatted at 2026-09-23 20:10:23 on dist-test-slave-th4g"
server_key: "3fe6accd84eeac3a371c29bb66fb20c4"
server_key_iv: "e6adbe3548d9e268f3bb5e6c8453a4d0"
server_key_version: "encryptionkey@0"
I20260923 20:10:23.900287 12274 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790194121577027-1706-0/minicluster-data/ts-1/wal/instance:
uuid: "8579308d6fd543d4bb9b3470d794dbf5"
format_stamp: "Formatted at 2026-09-23 20:10:23 on dist-test-slave-th4g"
server_key: "3fe6accd84eeac3a371c29bb66fb20c4"
server_key_iv: "e6adbe3548d9e268f3bb5e6c8453a4d0"
server_key_version: "encryptionkey@0"
I20260923 20:10:23.906090 12274 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260923 20:10:23.910038 12289 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:10:23.911698 12274 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260923 20:10:23.911933 12274 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790194121577027-1706-0/minicluster-data/ts-1/data,/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790194121577027-1706-0/minicluster-data/ts-1/wal
uuid: "8579308d6fd543d4bb9b3470d794dbf5"
format_stamp: "Formatted at 2026-09-23 20:10:23 on dist-test-slave-th4g"
server_key: "3fe6accd84eeac3a371c29bb66fb20c4"
server_key_iv: "e6adbe3548d9e268f3bb5e6c8453a4d0"
server_key_version: "encryptionkey@0"
I20260923 20:10:23.912174 12274 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790194121577027-1706-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790194121577027-1706-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790194121577027-1706-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:10:23.930003 12274 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:10:23.934762 12274 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:10:23.935070 12274 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:10:23.936065 12274 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 20:10:23.937695 12274 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 20:10:23.937805 12274 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 20:10:23.938016 12274 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 20:10:23.938086 12274 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 20:10:23.974040 12274 rpc_server.cc:307] RPC server started. Bound to: 127.1.170.130:42041
I20260923 20:10:23.974097 12401 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.170.130:42041 every 8 connection(s)
I20260923 20:10:23.975351 12274 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790194121577027-1706-0/minicluster-data/ts-1/data/info.pb
I20260923 20:10:23.981323 1706 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqUuku6/build/asan/bin/kudu as pid 12274
I20260923 20:10:23.981602 1706 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790194121577027-1706-0/minicluster-data/ts-1/wal/instance
I20260923 20:10:23.982265 1706 external_mini_cluster.cc:1428] Setting key 15cc86e7aec486101d3603914cd10aee
Sep 23 20:10:23 dist-test-slave-th4g krb5kdc[12034](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194223, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.170.130@KRBTEST.COM for kudu/127.1.170.190@KRBTEST.COM
I20260923 20:10:24.003087 12404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:10:23.978072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.170.130:47351 (local address 127.1.170.190:34019)
0923 20:10:23.978640 (+ 568us) server_negotiation.cc:206] Beginning negotiation
0923 20:10:23.978645 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 20:10:23.980341 (+ 1696us) server_negotiation.cc:407] Connection header received
0923 20:10:23.983451 (+ 3110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:10:23.983457 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:10:23.983558 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:10:23.983692 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:10:23.985609 (+ 1917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:10:23.986585 (+ 976us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:10:23.989516 (+ 2931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:10:23.990307 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:10:23.990457 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:10:23.995090 (+ 4633us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:10:23.995096 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:10:23.995099 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:10:23.995250 (+ 151us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:10:23.998907 (+ 3657us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:10:24.000149 (+ 1242us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:10:24.000155 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:10:24.000157 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:10:24.000336 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:10:24.000936 (+ 600us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:10:24.000940 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:10:24.000942 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:10:24.001389 (+ 447us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:10:24.001561 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0923 20:10:24.002587 (+ 1026us) server_negotiation.cc:299] Negotiation successful
0923 20:10:24.002802 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":403,"thread_start_us":300,"threads_started":1}
I20260923 20:10:24.004814 12405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:10:23.978706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.190:34019 (local address 127.1.170.130:47351)
0923 20:10:23.980089 (+ 1383us) negotiation.cc:107] Waiting for socket to connect
0923 20:10:23.980165 (+ 76us) client_negotiation.cc:174] Beginning negotiation
0923 20:10:23.983061 (+ 2896us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:10:23.983970 (+ 909us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:10:23.983979 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:10:23.984541 (+ 562us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:10:23.985394 (+ 853us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:10:23.985423 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:10:23.986789 (+ 1366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:10:23.986798 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:10:23.989279 (+ 2481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:10:23.989298 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:10:23.990570 (+ 1272us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:10:23.990580 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:10:23.990908 (+ 328us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:10:23.991740 (+ 832us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:10:23.991778 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:10:23.994779 (+ 3001us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:10:23.999163 (+ 4384us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:10:23.999176 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:10:23.999199 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:10:23.999936 (+ 737us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:10:24.000489 (+ 553us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:10:24.000497 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:10:24.000503 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:10:24.000790 (+ 287us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:10:24.001606 (+ 816us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:10:24.001617 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:10:24.002191 (+ 574us) client_negotiation.cc:769] Sending connection context
0923 20:10:24.002527 (+ 336us) client_negotiation.cc:240] Negotiation successful
0923 20:10:24.003218 (+ 691us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":836,"thread_start_us":351,"threads_started":1}
I20260923 20:10:24.006599 12402 heartbeater.cc:347] Connected to a master server at 127.1.170.190:34019
I20260923 20:10:24.006942 12402 heartbeater.cc:464] Registering TS with master...
I20260923 20:10:24.007802 12402 heartbeater.cc:511] Master 127.1.170.190:34019 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.1.170.131@KRBTEST.COM; defaulting to no policy
I20260923 20:10:24.009464 12079 ts_manager.cc:194] Registered new tserver with Master: 8579308d6fd543d4bb9b3470d794dbf5 (127.1.170.130:42041)
I20260923 20:10:24.010392 12079 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.170.130@KRBTEST.COM'} at 127.1.170.130:47351
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.170.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.170.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790194121577027-1706-0/krb5kdc/kudu_127.1.170.131.keytab.
Entry for principal kudu/127.1.170.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790194121577027-1706-0/krb5kdc/kudu_127.1.170.131.keytab.
I20260923 20:10:24.042390 1706 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqUuku6/build/asan/bin/kudu
/tmp/dist-test-taskqUuku6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790194121577027-1706-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790194121577027-1706-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790194121577027-1706-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790194121577027-1706-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.170.131:0
--local_ip_for_outbound_sockets=127.1.170.131
--webserver_interface=127.1.170.131
--webserver_port=0
--tserver_master_addrs=127.1.170.190:34019
--keytab_file=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790194121577027-1706-0/krb5kdc/kudu_127.1.170.131.keytab
--principal=kudu/127.1.170.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.170.148:34487
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790194121577027-1706-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790194121577027-1706-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790194121577027-1706-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:10:24.388211 12408 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:10:24.388583 12408 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:10:24.388765 12408 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:10:24.398775 12408 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:10:24.398978 12408 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:10:24.399190 12408 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.170.131
I20260923 20:10:24.411409 12408 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.170.148:34487
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790194121577027-1706-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790194121577027-1706-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790194121577027-1706-0/krb5kdc/kudu_127.1.170.131.keytab
--principal=kudu/127.1.170.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.170.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790194121577027-1706-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.170.131
--webserver_port=0
--tserver_master_addrs=127.1.170.190:34019
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.170.131
--log_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790194121577027-1706-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:10:24.413321 12408 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:10:24.415197 12408 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:10:24.432711 12413 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:10:24.432682 12414 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:10:24.432659 12416 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:10:24.433146 12408 server_base.cc:1034] running on GCE node
I20260923 20:10:24.434161 12408 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:10:24.435292 12408 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:10:24.436663 12408 hybrid_clock.cc:648] HybridClock initialized: now 1790194224436568 us; error 88 us; skew 500 ppm
I20260923 20:10:24.437034 12408 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:10:24 dist-test-slave-th4g krb5kdc[12034](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194224, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.170.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:10:24.440975 12408 init.cc:377] Logged in from keytab as kudu/127.1.170.131@KRBTEST.COM (short username kudu)
I20260923 20:10:24.442795 12408 webserver.cc:460] Webserver started at http://127.1.170.131:39553/ using document root <none> and password file <none>
I20260923 20:10:24.443573 12408 fs_manager.cc:359] Metadata directory not provided
I20260923 20:10:24.443743 12408 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:10:24.444079 12408 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:10:24.447286 12408 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790194121577027-1706-0/minicluster-data/ts-2/data/instance:
uuid: "15bbc58407634f919757157f8fbf4ee4"
format_stamp: "Formatted at 2026-09-23 20:10:24 on dist-test-slave-th4g"
server_key: "69ad4d3e6310582473971bf6b44a3e69"
server_key_iv: "35c8fe49552b960bd4d9df7df99e7918"
server_key_version: "encryptionkey@0"
I20260923 20:10:24.448184 12408 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790194121577027-1706-0/minicluster-data/ts-2/wal/instance:
uuid: "15bbc58407634f919757157f8fbf4ee4"
format_stamp: "Formatted at 2026-09-23 20:10:24 on dist-test-slave-th4g"
server_key: "69ad4d3e6310582473971bf6b44a3e69"
server_key_iv: "35c8fe49552b960bd4d9df7df99e7918"
server_key_version: "encryptionkey@0"
I20260923 20:10:24.454785 12408 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260923 20:10:24.459215 12423 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:10:24.461006 12408 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260923 20:10:24.461236 12408 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790194121577027-1706-0/minicluster-data/ts-2/data,/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790194121577027-1706-0/minicluster-data/ts-2/wal
uuid: "15bbc58407634f919757157f8fbf4ee4"
format_stamp: "Formatted at 2026-09-23 20:10:24 on dist-test-slave-th4g"
server_key: "69ad4d3e6310582473971bf6b44a3e69"
server_key_iv: "35c8fe49552b960bd4d9df7df99e7918"
server_key_version: "encryptionkey@0"
I20260923 20:10:24.461481 12408 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790194121577027-1706-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790194121577027-1706-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790194121577027-1706-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:10:24.480696 12268 heartbeater.cc:503] Master 127.1.170.190:34019 was elected leader, sending a full tablet report...
I20260923 20:10:24.481679 12408 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:10:24.487936 12408 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:10:24.488494 12408 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:10:24.489665 12408 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 20:10:24.491241 12408 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 20:10:24.491319 12408 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 20:10:24.491449 12408 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 20:10:24.491513 12408 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 20:10:24.533432 12408 rpc_server.cc:307] RPC server started. Bound to: 127.1.170.131:34829
I20260923 20:10:24.533437 12535 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.170.131:34829 every 8 connection(s)
I20260923 20:10:24.534986 12408 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790194121577027-1706-0/minicluster-data/ts-2/data/info.pb
I20260923 20:10:24.537492 1706 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqUuku6/build/asan/bin/kudu as pid 12408
I20260923 20:10:24.537627 1706 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790194121577027-1706-0/minicluster-data/ts-2/wal/instance
I20260923 20:10:24.538134 1706 external_mini_cluster.cc:1428] Setting key 43876714493a720e59bd31dc9e601443
Sep 23 20:10:24 dist-test-slave-th4g krb5kdc[12034](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194224, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.170.131@KRBTEST.COM for kudu/127.1.170.190@KRBTEST.COM
I20260923 20:10:24.562230 12538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:10:24.538685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.170.131:59693 (local address 127.1.170.190:34019)
0923 20:10:24.539295 (+ 610us) server_negotiation.cc:206] Beginning negotiation
0923 20:10:24.539300 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 20:10:24.540765 (+ 1465us) server_negotiation.cc:407] Connection header received
0923 20:10:24.543488 (+ 2723us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:10:24.543494 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:10:24.543596 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:10:24.543725 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:10:24.545605 (+ 1880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:10:24.546732 (+ 1127us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:10:24.548634 (+ 1902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:10:24.549062 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:10:24.549228 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:10:24.553355 (+ 4127us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:10:24.553362 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:10:24.553368 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:10:24.553419 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:10:24.557506 (+ 4087us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:10:24.558744 (+ 1238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:10:24.558749 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:10:24.558752 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:10:24.558947 (+ 195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:10:24.559644 (+ 697us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:10:24.559648 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:10:24.559650 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:10:24.560223 (+ 573us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:10:24.560405 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0923 20:10:24.561412 (+ 1007us) server_negotiation.cc:299] Negotiation successful
0923 20:10:24.561719 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":459,"thread_start_us":276,"threads_started":1}
I20260923 20:10:24.562852 12539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:10:24.539623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.190:34019 (local address 127.1.170.131:59693)
0923 20:10:24.540599 (+ 976us) negotiation.cc:107] Waiting for socket to connect
0923 20:10:24.540653 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0923 20:10:24.543160 (+ 2507us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:10:24.544086 (+ 926us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:10:24.544096 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:10:24.544787 (+ 691us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:10:24.545401 (+ 614us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:10:24.545419 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:10:24.546941 (+ 1522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:10:24.546946 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:10:24.548451 (+ 1505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:10:24.548464 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:10:24.549324 (+ 860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:10:24.549331 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:10:24.549615 (+ 284us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:10:24.550446 (+ 831us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:10:24.550471 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:10:24.553051 (+ 2580us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:10:24.557760 (+ 4709us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:10:24.557771 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:10:24.557792 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:10:24.558555 (+ 763us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:10:24.559162 (+ 607us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:10:24.559168 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:10:24.559172 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:10:24.559478 (+ 306us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:10:24.560429 (+ 951us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:10:24.560436 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:10:24.560843 (+ 407us) client_negotiation.cc:769] Sending connection context
0923 20:10:24.561104 (+ 261us) client_negotiation.cc:240] Negotiation successful
0923 20:10:24.561567 (+ 463us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":631,"thread_start_us":332,"threads_started":1}
I20260923 20:10:24.565030 12536 heartbeater.cc:347] Connected to a master server at 127.1.170.190:34019
I20260923 20:10:24.565438 12536 heartbeater.cc:464] Registering TS with master...
I20260923 20:10:24.566557 12536 heartbeater.cc:511] Master 127.1.170.190:34019 requested a full tablet report, sending...
I20260923 20:10:24.568394 12079 ts_manager.cc:194] Registered new tserver with Master: 15bbc58407634f919757157f8fbf4ee4 (127.1.170.131:34829)
I20260923 20:10:24.569708 12079 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.170.131@KRBTEST.COM'} at 127.1.170.131:59693
I20260923 20:10:24.582386 1706 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260923 20:10:24.605054 12538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:10:24.587451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42808 (local address 127.1.170.190:34019)
0923 20:10:24.587637 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0923 20:10:24.587642 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 20:10:24.587986 (+ 344us) server_negotiation.cc:407] Connection header received
0923 20:10:24.588129 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:10:24.588133 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:10:24.588207 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:10:24.588312 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:10:24.590676 (+ 2364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:10:24.591558 (+ 882us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:10:24.592910 (+ 1352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:10:24.593379 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:10:24.593489 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:10:24.596877 (+ 3388us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:10:24.596884 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:10:24.596888 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:10:24.597054 (+ 166us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:10:24.601179 (+ 4125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:10:24.602503 (+ 1324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:10:24.602508 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:10:24.602512 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:10:24.602658 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:10:24.603238 (+ 580us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:10:24.603242 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:10:24.603244 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:10:24.603595 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:10:24.603814 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0923 20:10:24.604588 (+ 774us) server_negotiation.cc:299] Negotiation successful
0923 20:10:24.604789 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260923 20:10:24.610402 12079 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42808:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 20:10:24.618652 12079 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 20:10:24.656921 12553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:10:24.632549 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.130:42041 (local address 127.0.0.1:45348)
0923 20:10:24.647363 (+ 14814us) negotiation.cc:107] Waiting for socket to connect
0923 20:10:24.647386 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0923 20:10:24.647550 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:10:24.648814 (+ 1264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:10:24.648818 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:10:24.648858 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:10:24.649984 (+ 1126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:10:24.649992 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:10:24.651276 (+ 1284us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:10:24.651284 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:10:24.654170 (+ 2886us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:10:24.654186 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:10:24.655754 (+ 1568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:10:24.655761 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:10:24.655956 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:10:24.656018 (+ 62us) client_negotiation.cc:769] Sending connection context
0923 20:10:24.656153 (+ 135us) client_negotiation.cc:240] Negotiation successful
0923 20:10:24.656368 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14655,"mutex_wait_us":752,"spinlock_wait_cycles":2304,"thread_start_us":10817,"threads_started":1}
I20260923 20:10:24.657326 12551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:10:24.632729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45348 (local address 127.1.170.130:42041)
0923 20:10:24.646609 (+ 13880us) server_negotiation.cc:206] Beginning negotiation
0923 20:10:24.646624 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0923 20:10:24.647698 (+ 1074us) server_negotiation.cc:407] Connection header received
0923 20:10:24.647831 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:10:24.648251 (+ 420us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:10:24.648454 (+ 203us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:10:24.648620 (+ 166us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:10:24.650127 (+ 1507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:10:24.651132 (+ 1005us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:10:24.654362 (+ 3230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:10:24.655538 (+ 1176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:10:24.655695 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:10:24.655909 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0923 20:10:24.656814 (+ 905us) server_negotiation.cc:299] Negotiation successful
0923 20:10:24.657011 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13635,"thread_start_us":13866,"threads_started":1}
I20260923 20:10:24.657723 12549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:10:24.634765 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.129:39829 (local address 127.0.0.1:54188)
0923 20:10:24.647656 (+ 12891us) negotiation.cc:107] Waiting for socket to connect
0923 20:10:24.647673 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0923 20:10:24.647778 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:10:24.648792 (+ 1014us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:10:24.648801 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:10:24.648831 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:10:24.649352 (+ 521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:10:24.649365 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:10:24.651694 (+ 2329us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:10:24.651699 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:10:24.654163 (+ 2464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:10:24.654178 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:10:24.656063 (+ 1885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:10:24.656069 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:10:24.656250 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:10:24.656279 (+ 29us) client_negotiation.cc:769] Sending connection context
0923 20:10:24.656361 (+ 82us) client_negotiation.cc:240] Negotiation successful
0923 20:10:24.656478 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12758,"mutex_wait_us":69,"thread_start_us":12279,"threads_started":1}
I20260923 20:10:24.658450 12550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:10:24.632487 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.131:34829 (local address 127.0.0.1:48316)
0923 20:10:24.643797 (+ 11310us) negotiation.cc:107] Waiting for socket to connect
0923 20:10:24.643848 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0923 20:10:24.644052 (+ 204us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:10:24.649243 (+ 5191us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:10:24.649252 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:10:24.649284 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:10:24.649677 (+ 393us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:10:24.649686 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:10:24.651870 (+ 2184us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:10:24.651876 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:10:24.654750 (+ 2874us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:10:24.654763 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:10:24.657096 (+ 2333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:10:24.657103 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:10:24.657270 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:10:24.657313 (+ 43us) client_negotiation.cc:769] Sending connection context
0923 20:10:24.657399 (+ 86us) client_negotiation.cc:240] Negotiation successful
0923 20:10:24.657743 (+ 344us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11033,"mutex_wait_us":3,"thread_start_us":14889,"threads_started":1}
I20260923 20:10:24.658975 12554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:10:24.635045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54188 (local address 127.1.170.129:39829)
0923 20:10:24.643594 (+ 8549us) server_negotiation.cc:206] Beginning negotiation
0923 20:10:24.643611 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0923 20:10:24.647886 (+ 4275us) server_negotiation.cc:407] Connection header received
0923 20:10:24.647976 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:10:24.647984 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:10:24.648285 (+ 301us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:10:24.648495 (+ 210us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:10:24.649609 (+ 1114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:10:24.651527 (+ 1918us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:10:24.654346 (+ 2819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:10:24.655804 (+ 1458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:10:24.655956 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:10:24.656218 (+ 262us) server_negotiation.cc:1035] Waiting for connection context
0923 20:10:24.658370 (+ 2152us) server_negotiation.cc:299] Negotiation successful
0923 20:10:24.658597 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7637,"thread_start_us":7457,"threads_started":1}
I20260923 20:10:24.659363 12552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:10:24.632832 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48316 (local address 127.1.170.131:34829)
0923 20:10:24.648237 (+ 15405us) server_negotiation.cc:206] Beginning negotiation
0923 20:10:24.648247 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0923 20:10:24.648280 (+ 33us) server_negotiation.cc:407] Connection header received
0923 20:10:24.648381 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:10:24.648388 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:10:24.648628 (+ 240us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:10:24.648859 (+ 231us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:10:24.649865 (+ 1006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:10:24.651617 (+ 1752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:10:24.654925 (+ 3308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:10:24.656572 (+ 1647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:10:24.656728 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:10:24.656952 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0923 20:10:24.658784 (+ 1832us) server_negotiation.cc:299] Negotiation successful
0923 20:10:24.658967 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15207,"thread_start_us":15048,"threads_started":1}
I20260923 20:10:24.662552 12471 tablet_service.cc:1467] Processing CreateTablet for tablet a60fbd348d6f4fef8741529739a442c6 (DEFAULT_TABLE table=test-table [id=70ae311721c2403f86d4a07c813fd30a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 20:10:24.662551 12337 tablet_service.cc:1467] Processing CreateTablet for tablet a60fbd348d6f4fef8741529739a442c6 (DEFAULT_TABLE table=test-table [id=70ae311721c2403f86d4a07c813fd30a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 20:10:24.662576 12203 tablet_service.cc:1467] Processing CreateTablet for tablet a60fbd348d6f4fef8741529739a442c6 (DEFAULT_TABLE table=test-table [id=70ae311721c2403f86d4a07c813fd30a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 20:10:24.664068 12203 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a60fbd348d6f4fef8741529739a442c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:10:24.664089 12337 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a60fbd348d6f4fef8741529739a442c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:10:24.664489 12471 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a60fbd348d6f4fef8741529739a442c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:10:24.678993 12555 tablet_bootstrap.cc:492] T a60fbd348d6f4fef8741529739a442c6 P 15bbc58407634f919757157f8fbf4ee4: Bootstrap starting.
I20260923 20:10:24.682682 12557 tablet_bootstrap.cc:492] T a60fbd348d6f4fef8741529739a442c6 P 1776720c83cc42a38935643fb249c406: Bootstrap starting.
I20260923 20:10:24.682900 12555 tablet_bootstrap.cc:654] T a60fbd348d6f4fef8741529739a442c6 P 15bbc58407634f919757157f8fbf4ee4: Neither blocks nor log segments found. Creating new log.
I20260923 20:10:24.684456 12555 log.cc:824] T a60fbd348d6f4fef8741529739a442c6 P 15bbc58407634f919757157f8fbf4ee4: Log is configured to *not* fsync() on all Append() calls
I20260923 20:10:24.687209 12556 tablet_bootstrap.cc:492] T a60fbd348d6f4fef8741529739a442c6 P 8579308d6fd543d4bb9b3470d794dbf5: Bootstrap starting.
I20260923 20:10:24.687376 12557 tablet_bootstrap.cc:654] T a60fbd348d6f4fef8741529739a442c6 P 1776720c83cc42a38935643fb249c406: Neither blocks nor log segments found. Creating new log.
I20260923 20:10:24.688740 12557 log.cc:824] T a60fbd348d6f4fef8741529739a442c6 P 1776720c83cc42a38935643fb249c406: Log is configured to *not* fsync() on all Append() calls
I20260923 20:10:24.689090 12555 tablet_bootstrap.cc:492] T a60fbd348d6f4fef8741529739a442c6 P 15bbc58407634f919757157f8fbf4ee4: No bootstrap required, opened a new log
I20260923 20:10:24.689445 12555 ts_tablet_manager.cc:1397] T a60fbd348d6f4fef8741529739a442c6 P 15bbc58407634f919757157f8fbf4ee4: Time spent bootstrapping tablet: real 0.011s user 0.007s sys 0.000s
I20260923 20:10:24.690436 12556 tablet_bootstrap.cc:654] T a60fbd348d6f4fef8741529739a442c6 P 8579308d6fd543d4bb9b3470d794dbf5: Neither blocks nor log segments found. Creating new log.
I20260923 20:10:24.691416 12556 log.cc:824] T a60fbd348d6f4fef8741529739a442c6 P 8579308d6fd543d4bb9b3470d794dbf5: Log is configured to *not* fsync() on all Append() calls
I20260923 20:10:24.692104 12557 tablet_bootstrap.cc:492] T a60fbd348d6f4fef8741529739a442c6 P 1776720c83cc42a38935643fb249c406: No bootstrap required, opened a new log
I20260923 20:10:24.692400 12557 ts_tablet_manager.cc:1397] T a60fbd348d6f4fef8741529739a442c6 P 1776720c83cc42a38935643fb249c406: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260923 20:10:24.694010 12556 tablet_bootstrap.cc:492] T a60fbd348d6f4fef8741529739a442c6 P 8579308d6fd543d4bb9b3470d794dbf5: No bootstrap required, opened a new log
I20260923 20:10:24.694271 12556 ts_tablet_manager.cc:1397] T a60fbd348d6f4fef8741529739a442c6 P 8579308d6fd543d4bb9b3470d794dbf5: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260923 20:10:24.701609 12556 raft_consensus.cc:348] T a60fbd348d6f4fef8741529739a442c6 P 8579308d6fd543d4bb9b3470d794dbf5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15bbc58407634f919757157f8fbf4ee4" member_type: VOTER last_known_addr { host: "127.1.170.131" port: 34829 } } peers { permanent_uuid: "8579308d6fd543d4bb9b3470d794dbf5" member_type: VOTER last_known_addr { host: "127.1.170.130" port: 42041 } } peers { permanent_uuid: "1776720c83cc42a38935643fb249c406" member_type: VOTER last_known_addr { host: "127.1.170.129" port: 39829 } }
I20260923 20:10:24.701604 12555 raft_consensus.cc:348] T a60fbd348d6f4fef8741529739a442c6 P 15bbc58407634f919757157f8fbf4ee4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15bbc58407634f919757157f8fbf4ee4" member_type: VOTER last_known_addr { host: "127.1.170.131" port: 34829 } } peers { permanent_uuid: "8579308d6fd543d4bb9b3470d794dbf5" member_type: VOTER last_known_addr { host: "127.1.170.130" port: 42041 } } peers { permanent_uuid: "1776720c83cc42a38935643fb249c406" member_type: VOTER last_known_addr { host: "127.1.170.129" port: 39829 } }
I20260923 20:10:24.702129 12556 raft_consensus.cc:374] T a60fbd348d6f4fef8741529739a442c6 P 8579308d6fd543d4bb9b3470d794dbf5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:10:24.702145 12555 raft_consensus.cc:374] T a60fbd348d6f4fef8741529739a442c6 P 15bbc58407634f919757157f8fbf4ee4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:10:24.702289 12556 raft_consensus.cc:729] T a60fbd348d6f4fef8741529739a442c6 P 8579308d6fd543d4bb9b3470d794dbf5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8579308d6fd543d4bb9b3470d794dbf5, State: Initialized, Role: FOLLOWER
I20260923 20:10:24.702323 12555 raft_consensus.cc:729] T a60fbd348d6f4fef8741529739a442c6 P 15bbc58407634f919757157f8fbf4ee4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 15bbc58407634f919757157f8fbf4ee4, State: Initialized, Role: FOLLOWER
I20260923 20:10:24.702986 12556 consensus_queue.cc:260] T a60fbd348d6f4fef8741529739a442c6 P 8579308d6fd543d4bb9b3470d794dbf5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15bbc58407634f919757157f8fbf4ee4" member_type: VOTER last_known_addr { host: "127.1.170.131" port: 34829 } } peers { permanent_uuid: "8579308d6fd543d4bb9b3470d794dbf5" member_type: VOTER last_known_addr { host: "127.1.170.130" port: 42041 } } peers { permanent_uuid: "1776720c83cc42a38935643fb249c406" member_type: VOTER last_known_addr { host: "127.1.170.129" port: 39829 } }
I20260923 20:10:24.703078 12555 consensus_queue.cc:260] T a60fbd348d6f4fef8741529739a442c6 P 15bbc58407634f919757157f8fbf4ee4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15bbc58407634f919757157f8fbf4ee4" member_type: VOTER last_known_addr { host: "127.1.170.131" port: 34829 } } peers { permanent_uuid: "8579308d6fd543d4bb9b3470d794dbf5" member_type: VOTER last_known_addr { host: "127.1.170.130" port: 42041 } } peers { permanent_uuid: "1776720c83cc42a38935643fb249c406" member_type: VOTER last_known_addr { host: "127.1.170.129" port: 39829 } }
I20260923 20:10:24.704270 12402 heartbeater.cc:503] Master 127.1.170.190:34019 was elected leader, sending a full tablet report...
I20260923 20:10:24.704753 12556 ts_tablet_manager.cc:1428] T a60fbd348d6f4fef8741529739a442c6 P 8579308d6fd543d4bb9b3470d794dbf5: Time spent starting tablet: real 0.010s user 0.006s sys 0.004s
I20260923 20:10:24.705679 12557 raft_consensus.cc:348] T a60fbd348d6f4fef8741529739a442c6 P 1776720c83cc42a38935643fb249c406 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15bbc58407634f919757157f8fbf4ee4" member_type: VOTER last_known_addr { host: "127.1.170.131" port: 34829 } } peers { permanent_uuid: "8579308d6fd543d4bb9b3470d794dbf5" member_type: VOTER last_known_addr { host: "127.1.170.130" port: 42041 } } peers { permanent_uuid: "1776720c83cc42a38935643fb249c406" member_type: VOTER last_known_addr { host: "127.1.170.129" port: 39829 } }
I20260923 20:10:24.706489 12557 raft_consensus.cc:374] T a60fbd348d6f4fef8741529739a442c6 P 1776720c83cc42a38935643fb249c406 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:10:24.706645 12557 raft_consensus.cc:729] T a60fbd348d6f4fef8741529739a442c6 P 1776720c83cc42a38935643fb249c406 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1776720c83cc42a38935643fb249c406, State: Initialized, Role: FOLLOWER
I20260923 20:10:24.707536 12557 consensus_queue.cc:260] T a60fbd348d6f4fef8741529739a442c6 P 1776720c83cc42a38935643fb249c406 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15bbc58407634f919757157f8fbf4ee4" member_type: VOTER last_known_addr { host: "127.1.170.131" port: 34829 } } peers { permanent_uuid: "8579308d6fd543d4bb9b3470d794dbf5" member_type: VOTER last_known_addr { host: "127.1.170.130" port: 42041 } } peers { permanent_uuid: "1776720c83cc42a38935643fb249c406" member_type: VOTER last_known_addr { host: "127.1.170.129" port: 39829 } }
I20260923 20:10:24.710058 12557 ts_tablet_manager.cc:1428] T a60fbd348d6f4fef8741529739a442c6 P 1776720c83cc42a38935643fb249c406: Time spent starting tablet: real 0.018s user 0.019s sys 0.000s
I20260923 20:10:24.711265 12536 heartbeater.cc:503] Master 127.1.170.190:34019 was elected leader, sending a full tablet report...
I20260923 20:10:24.711949 12555 ts_tablet_manager.cc:1428] T a60fbd348d6f4fef8741529739a442c6 P 15bbc58407634f919757157f8fbf4ee4: Time spent starting tablet: real 0.022s user 0.013s sys 0.008s
W20260923 20:10:24.727586 12403 tablet.cc:2369] T a60fbd348d6f4fef8741529739a442c6 P 8579308d6fd543d4bb9b3470d794dbf5: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260923 20:10:24.786967 12537 tablet.cc:2369] T a60fbd348d6f4fef8741529739a442c6 P 15bbc58407634f919757157f8fbf4ee4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 20:10:24.826263 12563 raft_consensus.cc:482] T a60fbd348d6f4fef8741529739a442c6 P 1776720c83cc42a38935643fb249c406 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 20:10:24.826565 12563 raft_consensus.cc:504] T a60fbd348d6f4fef8741529739a442c6 P 1776720c83cc42a38935643fb249c406 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15bbc58407634f919757157f8fbf4ee4" member_type: VOTER last_known_addr { host: "127.1.170.131" port: 34829 } } peers { permanent_uuid: "8579308d6fd543d4bb9b3470d794dbf5" member_type: VOTER last_known_addr { host: "127.1.170.130" port: 42041 } } peers { permanent_uuid: "1776720c83cc42a38935643fb249c406" member_type: VOTER last_known_addr { host: "127.1.170.129" port: 39829 } }
I20260923 20:10:24.828917 12563 leader_election.cc:288] T a60fbd348d6f4fef8741529739a442c6 P 1776720c83cc42a38935643fb249c406 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 15bbc58407634f919757157f8fbf4ee4 (127.1.170.131:34829), 8579308d6fd543d4bb9b3470d794dbf5 (127.1.170.130:42041)
I20260923 20:10:24.839012 12551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:10:24.830088 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.170.129:58563 (local address 127.1.170.130:42041)
0923 20:10:24.830384 (+ 296us) server_negotiation.cc:206] Beginning negotiation
0923 20:10:24.830392 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 20:10:24.831490 (+ 1098us) server_negotiation.cc:407] Connection header received
0923 20:10:24.831808 (+ 318us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:10:24.831814 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:10:24.831892 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:10:24.832001 (+ 109us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:10:24.832692 (+ 691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:10:24.834012 (+ 1320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:10:24.836669 (+ 2657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:10:24.837967 (+ 1298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:10:24.838135 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:10:24.838219 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0923 20:10:24.838542 (+ 323us) server_negotiation.cc:299] Negotiation successful
0923 20:10:24.838693 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121}
I20260923 20:10:24.839156 12564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:10:24.829672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.130:42041 (local address 127.1.170.129:58563)
0923 20:10:24.831330 (+ 1658us) negotiation.cc:107] Waiting for socket to connect
0923 20:10:24.831359 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0923 20:10:24.831639 (+ 280us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:10:24.832022 (+ 383us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:10:24.832026 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:10:24.832046 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:10:24.832539 (+ 493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:10:24.832549 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:10:24.834153 (+ 1604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:10:24.834159 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:10:24.836486 (+ 2327us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:10:24.836500 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:10:24.838169 (+ 1669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:10:24.838176 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:10:24.838350 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:10:24.838385 (+ 35us) client_negotiation.cc:769] Sending connection context
0923 20:10:24.838489 (+ 104us) client_negotiation.cc:240] Negotiation successful
0923 20:10:24.838686 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1495,"mutex_wait_us":152,"thread_start_us":1265,"threads_started":1}
I20260923 20:10:24.839140 12552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:10:24.830715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.170.129:42715 (local address 127.1.170.131:34829)
0923 20:10:24.830909 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0923 20:10:24.830916 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:10:24.831838 (+ 922us) server_negotiation.cc:407] Connection header received
0923 20:10:24.832003 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:10:24.832008 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:10:24.832084 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:10:24.832182 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:10:24.832861 (+ 679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:10:24.834113 (+ 1252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:10:24.836431 (+ 2318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:10:24.837957 (+ 1526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:10:24.838136 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:10:24.838212 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0923 20:10:24.838739 (+ 527us) server_negotiation.cc:299] Negotiation successful
0923 20:10:24.838888 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260923 20:10:24.839259 12565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:10:24.829745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.131:34829 (local address 127.1.170.129:42715)
0923 20:10:24.831732 (+ 1987us) negotiation.cc:107] Waiting for socket to connect
0923 20:10:24.831750 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0923 20:10:24.831866 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:10:24.832216 (+ 350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:10:24.832221 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:10:24.832249 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:10:24.832686 (+ 437us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:10:24.832696 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:10:24.834255 (+ 1559us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:10:24.834262 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:10:24.836212 (+ 1950us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:10:24.836226 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:10:24.838379 (+ 2153us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:10:24.838384 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:10:24.838551 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:10:24.838581 (+ 30us) client_negotiation.cc:769] Sending connection context
0923 20:10:24.838672 (+ 91us) client_negotiation.cc:240] Negotiation successful
0923 20:10:24.838804 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1855,"mutex_wait_us":114,"thread_start_us":1613,"threads_started":1}
I20260923 20:10:24.840493 12491 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a60fbd348d6f4fef8741529739a442c6" candidate_uuid: "1776720c83cc42a38935643fb249c406" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "15bbc58407634f919757157f8fbf4ee4" is_pre_election: true
I20260923 20:10:24.840816 12357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a60fbd348d6f4fef8741529739a442c6" candidate_uuid: "1776720c83cc42a38935643fb249c406" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8579308d6fd543d4bb9b3470d794dbf5" is_pre_election: true
I20260923 20:10:24.841144 12491 raft_consensus.cc:2454] T a60fbd348d6f4fef8741529739a442c6 P 15bbc58407634f919757157f8fbf4ee4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1776720c83cc42a38935643fb249c406 in term 0.
I20260923 20:10:24.841368 12357 raft_consensus.cc:2454] T a60fbd348d6f4fef8741529739a442c6 P 8579308d6fd543d4bb9b3470d794dbf5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1776720c83cc42a38935643fb249c406 in term 0.
I20260923 20:10:24.841957 12158 leader_election.cc:302] T a60fbd348d6f4fef8741529739a442c6 P 1776720c83cc42a38935643fb249c406 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 15bbc58407634f919757157f8fbf4ee4, 1776720c83cc42a38935643fb249c406; no voters:
I20260923 20:10:24.842384 12563 raft_consensus.cc:2781] T a60fbd348d6f4fef8741529739a442c6 P 1776720c83cc42a38935643fb249c406 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 20:10:24.842526 12563 raft_consensus.cc:482] T a60fbd348d6f4fef8741529739a442c6 P 1776720c83cc42a38935643fb249c406 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 20:10:24.842672 12563 raft_consensus.cc:3037] T a60fbd348d6f4fef8741529739a442c6 P 1776720c83cc42a38935643fb249c406 [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:10:24.844897 12563 raft_consensus.cc:504] T a60fbd348d6f4fef8741529739a442c6 P 1776720c83cc42a38935643fb249c406 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15bbc58407634f919757157f8fbf4ee4" member_type: VOTER last_known_addr { host: "127.1.170.131" port: 34829 } } peers { permanent_uuid: "8579308d6fd543d4bb9b3470d794dbf5" member_type: VOTER last_known_addr { host: "127.1.170.130" port: 42041 } } peers { permanent_uuid: "1776720c83cc42a38935643fb249c406" member_type: VOTER last_known_addr { host: "127.1.170.129" port: 39829 } }
I20260923 20:10:24.845558 12563 leader_election.cc:288] T a60fbd348d6f4fef8741529739a442c6 P 1776720c83cc42a38935643fb249c406 [CANDIDATE]: Term 1 election: Requested vote from peers 15bbc58407634f919757157f8fbf4ee4 (127.1.170.131:34829), 8579308d6fd543d4bb9b3470d794dbf5 (127.1.170.130:42041)
I20260923 20:10:24.846282 12491 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a60fbd348d6f4fef8741529739a442c6" candidate_uuid: "1776720c83cc42a38935643fb249c406" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "15bbc58407634f919757157f8fbf4ee4"
I20260923 20:10:24.846436 12357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a60fbd348d6f4fef8741529739a442c6" candidate_uuid: "1776720c83cc42a38935643fb249c406" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8579308d6fd543d4bb9b3470d794dbf5"
I20260923 20:10:24.846496 12491 raft_consensus.cc:3037] T a60fbd348d6f4fef8741529739a442c6 P 15bbc58407634f919757157f8fbf4ee4 [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:10:24.846683 12357 raft_consensus.cc:3037] T a60fbd348d6f4fef8741529739a442c6 P 8579308d6fd543d4bb9b3470d794dbf5 [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:10:24.848731 12491 raft_consensus.cc:2454] T a60fbd348d6f4fef8741529739a442c6 P 15bbc58407634f919757157f8fbf4ee4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1776720c83cc42a38935643fb249c406 in term 1.
I20260923 20:10:24.848762 12357 raft_consensus.cc:2454] T a60fbd348d6f4fef8741529739a442c6 P 8579308d6fd543d4bb9b3470d794dbf5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1776720c83cc42a38935643fb249c406 in term 1.
I20260923 20:10:24.849438 12158 leader_election.cc:302] T a60fbd348d6f4fef8741529739a442c6 P 1776720c83cc42a38935643fb249c406 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 15bbc58407634f919757157f8fbf4ee4, 1776720c83cc42a38935643fb249c406, 8579308d6fd543d4bb9b3470d794dbf5; no voters:
I20260923 20:10:24.849974 12563 raft_consensus.cc:2781] T a60fbd348d6f4fef8741529739a442c6 P 1776720c83cc42a38935643fb249c406 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 20:10:24.850555 12563 raft_consensus.cc:686] T a60fbd348d6f4fef8741529739a442c6 P 1776720c83cc42a38935643fb249c406 [term 1 LEADER]: Becoming Leader. State: Replica: 1776720c83cc42a38935643fb249c406, State: Running, Role: LEADER
I20260923 20:10:24.851279 12563 consensus_queue.cc:237] T a60fbd348d6f4fef8741529739a442c6 P 1776720c83cc42a38935643fb249c406 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15bbc58407634f919757157f8fbf4ee4" member_type: VOTER last_known_addr { host: "127.1.170.131" port: 34829 } } peers { permanent_uuid: "8579308d6fd543d4bb9b3470d794dbf5" member_type: VOTER last_known_addr { host: "127.1.170.130" port: 42041 } } peers { permanent_uuid: "1776720c83cc42a38935643fb249c406" member_type: VOTER last_known_addr { host: "127.1.170.129" port: 39829 } }
I20260923 20:10:24.856649 12079 catalog_manager.cc:5495] T a60fbd348d6f4fef8741529739a442c6 P 1776720c83cc42a38935643fb249c406 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1776720c83cc42a38935643fb249c406 (127.1.170.129). New cstate: current_term: 1 leader_uuid: "1776720c83cc42a38935643fb249c406" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15bbc58407634f919757157f8fbf4ee4" member_type: VOTER last_known_addr { host: "127.1.170.131" port: 34829 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8579308d6fd543d4bb9b3470d794dbf5" member_type: VOTER last_known_addr { host: "127.1.170.130" port: 42041 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1776720c83cc42a38935643fb249c406" member_type: VOTER last_known_addr { host: "127.1.170.129" port: 39829 } health_report { overall_health: HEALTHY } } }
I20260923 20:10:24.903136 12554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:10:24.895916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54192 (local address 127.1.170.129:39829)
0923 20:10:24.896136 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0923 20:10:24.896143 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:10:24.896163 (+ 20us) server_negotiation.cc:407] Connection header received
0923 20:10:24.896408 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:10:24.896412 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:10:24.896512 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:10:24.896682 (+ 170us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 20:10:24.897094 (+ 412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:10:24.897867 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:10:24.899797 (+ 1930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:10:24.900350 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:10:24.900477 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:10:24.901421 (+ 944us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 20:10:24.902251 (+ 830us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 20:10:24.902412 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0923 20:10:24.902636 (+ 224us) server_negotiation.cc:299] Negotiation successful
0923 20:10:24.902738 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260923 20:10:24.912443 12491 raft_consensus.cc:1261] T a60fbd348d6f4fef8741529739a442c6 P 15bbc58407634f919757157f8fbf4ee4 [term 1 FOLLOWER]: Refusing update from remote peer 1776720c83cc42a38935643fb249c406: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 20:10:24.912832 12357 raft_consensus.cc:1261] T a60fbd348d6f4fef8741529739a442c6 P 8579308d6fd543d4bb9b3470d794dbf5 [term 1 FOLLOWER]: Refusing update from remote peer 1776720c83cc42a38935643fb249c406: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 20:10:24.913596 12566 consensus_queue.cc:1035] T a60fbd348d6f4fef8741529739a442c6 P 1776720c83cc42a38935643fb249c406 [LEADER]: Connected to new peer: Peer: permanent_uuid: "15bbc58407634f919757157f8fbf4ee4" member_type: VOTER last_known_addr { host: "127.1.170.131" port: 34829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 20:10:24.914197 12563 consensus_queue.cc:1035] T a60fbd348d6f4fef8741529739a442c6 P 1776720c83cc42a38935643fb249c406 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8579308d6fd543d4bb9b3470d794dbf5" member_type: VOTER last_known_addr { host: "127.1.170.130" port: 42041 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 20:10:24.925060 12570 mvcc.cc:205] Tried to move back new op lower bound from 7332635545230417920 to 7332635544995995648. Current Snapshot: MvccSnapshot[applied={T|T < 7332635545230417920}]
I20260923 20:10:24.938552 12078 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42808:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 20:10:24.938894 12078 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42808:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 20:10:24.943462 12078 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0e40b98f6ac647018944146aecfb233a: Sending DeleteTablet for 3 replicas of tablet a60fbd348d6f4fef8741529739a442c6
I20260923 20:10:24.944631 12471 tablet_service.cc:1514] Processing DeleteTablet for tablet a60fbd348d6f4fef8741529739a442c6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 20:10:24 UTC) from {username='kudu', principal='kudu/127.1.170.190@KRBTEST.COM'} at 127.0.0.1:48316
I20260923 20:10:24.944638 12203 tablet_service.cc:1514] Processing DeleteTablet for tablet a60fbd348d6f4fef8741529739a442c6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 20:10:24 UTC) from {username='kudu', principal='kudu/127.1.170.190@KRBTEST.COM'} at 127.0.0.1:54188
I20260923 20:10:24.944938 12337 tablet_service.cc:1514] Processing DeleteTablet for tablet a60fbd348d6f4fef8741529739a442c6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 20:10:24 UTC) from {username='kudu', principal='kudu/127.1.170.190@KRBTEST.COM'} at 127.0.0.1:45348
I20260923 20:10:24.947104 1706 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskqUuku6/build/asan/bin/kudu pid 12140
I20260923 20:10:24.958622 12579 tablet_replica.cc:330] T a60fbd348d6f4fef8741529739a442c6 P 15bbc58407634f919757157f8fbf4ee4: stopping tablet replica
I20260923 20:10:24.959203 12579 raft_consensus.cc:2229] T a60fbd348d6f4fef8741529739a442c6 P 15bbc58407634f919757157f8fbf4ee4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 20:10:24.959872 12579 raft_consensus.cc:2258] T a60fbd348d6f4fef8741529739a442c6 P 15bbc58407634f919757157f8fbf4ee4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 20:10:24.962339 12579 ts_tablet_manager.cc:1905] T a60fbd348d6f4fef8741529739a442c6 P 15bbc58407634f919757157f8fbf4ee4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 20:10:24.965849 12580 tablet_replica.cc:330] T a60fbd348d6f4fef8741529739a442c6 P 8579308d6fd543d4bb9b3470d794dbf5: stopping tablet replica
I20260923 20:10:24.966006 12578 tablet_replica.cc:330] T a60fbd348d6f4fef8741529739a442c6 P 1776720c83cc42a38935643fb249c406: stopping tablet replica
I20260923 20:10:24.966537 12580 raft_consensus.cc:2229] T a60fbd348d6f4fef8741529739a442c6 P 8579308d6fd543d4bb9b3470d794dbf5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 20:10:24.966580 12578 raft_consensus.cc:2229] T a60fbd348d6f4fef8741529739a442c6 P 1776720c83cc42a38935643fb249c406 [term 1 LEADER]: Raft consensus shutting down.
I20260923 20:10:24.967135 12580 raft_consensus.cc:2258] T a60fbd348d6f4fef8741529739a442c6 P 8579308d6fd543d4bb9b3470d794dbf5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 20:10:24.967267 12578 raft_consensus.cc:2258] T a60fbd348d6f4fef8741529739a442c6 P 1776720c83cc42a38935643fb249c406 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 20:10:24.969488 12580 ts_tablet_manager.cc:1905] T a60fbd348d6f4fef8741529739a442c6 P 8579308d6fd543d4bb9b3470d794dbf5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 20:10:24.969497 12578 ts_tablet_manager.cc:1905] T a60fbd348d6f4fef8741529739a442c6 P 1776720c83cc42a38935643fb249c406: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 20:10:24.971681 12579 ts_tablet_manager.cc:1918] T a60fbd348d6f4fef8741529739a442c6 P 15bbc58407634f919757157f8fbf4ee4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 20:10:24.976670 12580 ts_tablet_manager.cc:1918] T a60fbd348d6f4fef8741529739a442c6 P 8579308d6fd543d4bb9b3470d794dbf5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 20:10:24.976903 12580 log.cc:1196] T a60fbd348d6f4fef8741529739a442c6 P 8579308d6fd543d4bb9b3470d794dbf5: Deleting WAL directory at /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790194121577027-1706-0/minicluster-data/ts-1/wal/wals/a60fbd348d6f4fef8741529739a442c6
I20260923 20:10:24.977216 12578 ts_tablet_manager.cc:1918] T a60fbd348d6f4fef8741529739a442c6 P 1776720c83cc42a38935643fb249c406: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 20:10:24.977480 12578 log.cc:1196] T a60fbd348d6f4fef8741529739a442c6 P 1776720c83cc42a38935643fb249c406: Deleting WAL directory at /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790194121577027-1706-0/minicluster-data/ts-0/wal/wals/a60fbd348d6f4fef8741529739a442c6
I20260923 20:10:24.977514 12580 ts_tablet_manager.cc:1939] T a60fbd348d6f4fef8741529739a442c6 P 8579308d6fd543d4bb9b3470d794dbf5: Deleting consensus metadata
I20260923 20:10:24.978134 12578 ts_tablet_manager.cc:1939] T a60fbd348d6f4fef8741529739a442c6 P 1776720c83cc42a38935643fb249c406: Deleting consensus metadata
I20260923 20:10:24.979688 12066 catalog_manager.cc:4841] TS 8579308d6fd543d4bb9b3470d794dbf5 (127.1.170.130:42041): tablet a60fbd348d6f4fef8741529739a442c6 (table test-table [id=70ae311721c2403f86d4a07c813fd30a]) successfully deleted
I20260923 20:10:24.979992 12064 catalog_manager.cc:4841] TS 1776720c83cc42a38935643fb249c406 (127.1.170.129:39829): tablet a60fbd348d6f4fef8741529739a442c6 (table test-table [id=70ae311721c2403f86d4a07c813fd30a]) successfully deleted
I20260923 20:10:24.983395 12579 log.cc:1196] T a60fbd348d6f4fef8741529739a442c6 P 15bbc58407634f919757157f8fbf4ee4: Deleting WAL directory at /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790194121577027-1706-0/minicluster-data/ts-2/wal/wals/a60fbd348d6f4fef8741529739a442c6
Sep 23 20:10:24 dist-test-slave-th4g krb5kdc[12034](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194222, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.170.129@KRBTEST.COM
I20260923 20:10:24.984069 12579 ts_tablet_manager.cc:1939] T a60fbd348d6f4fef8741529739a442c6 P 15bbc58407634f919757157f8fbf4ee4: Deleting consensus metadata
I20260923 20:10:24.987063 12065 catalog_manager.cc:4841] TS 15bbc58407634f919757157f8fbf4ee4 (127.1.170.131:34829): tablet a60fbd348d6f4fef8741529739a442c6 (table test-table [id=70ae311721c2403f86d4a07c813fd30a]) successfully deleted
I20260923 20:10:24.995217 12554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:10:24.949051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54208 (local address 127.1.170.129:39829)
0923 20:10:24.949289 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0923 20:10:24.949299 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0923 20:10:24.953791 (+ 4492us) server_negotiation.cc:407] Connection header received
0923 20:10:24.954546 (+ 755us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:10:24.954555 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:10:24.954667 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:10:24.954779 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:10:24.959046 (+ 4267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:10:24.960688 (+ 1642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:10:24.965714 (+ 5026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:10:24.966438 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:10:24.966564 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:10:24.985696 (+ 19132us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:10:24.985716 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:10:24.985738 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:10:24.985803 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:10:24.991214 (+ 5411us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:10:24.992328 (+ 1114us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:10:24.992342 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:10:24.992348 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:10:24.992621 (+ 273us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:10:24.993219 (+ 598us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:10:24.993226 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:10:24.993229 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:10:24.993827 (+ 598us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:10:24.994240 (+ 413us) server_negotiation.cc:1035] Waiting for connection context
0923 20:10:24.994541 (+ 301us) server_negotiation.cc:299] Negotiation successful
0923 20:10:24.994835 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260923 20:10:24.996021 12263 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:10:25.102137 1706 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqUuku6/build/asan/bin/kudu with pid 12140
I20260923 20:10:25.120800 1706 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskqUuku6/build/asan/bin/kudu pid 12274
Sep 23 20:10:25 dist-test-slave-th4g krb5kdc[12034](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194222, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.170.130@KRBTEST.COM
I20260923 20:10:25.135932 12551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:10:25.121623 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45360 (local address 127.1.170.130:42041)
0923 20:10:25.121859 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0923 20:10:25.121867 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 20:10:25.121968 (+ 101us) server_negotiation.cc:407] Connection header received
0923 20:10:25.122182 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:10:25.122186 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:10:25.122248 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:10:25.122388 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:10:25.123623 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:10:25.124358 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:10:25.125552 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:10:25.126010 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:10:25.126137 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:10:25.130143 (+ 4006us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:10:25.130156 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:10:25.130189 (+ 33us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:10:25.130223 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:10:25.132991 (+ 2768us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:10:25.133711 (+ 720us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:10:25.133720 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:10:25.133722 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:10:25.133847 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:10:25.134472 (+ 625us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:10:25.134476 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:10:25.134478 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:10:25.134890 (+ 412us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:10:25.135116 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0923 20:10:25.135512 (+ 396us) server_negotiation.cc:299] Negotiation successful
0923 20:10:25.135674 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117}
I20260923 20:10:25.136857 12397 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:10:25.226368 1706 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqUuku6/build/asan/bin/kudu with pid 12274
I20260923 20:10:25.244138 1706 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskqUuku6/build/asan/bin/kudu pid 12408
Sep 23 20:10:25 dist-test-slave-th4g krb5kdc[12034](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194222, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.170.131@KRBTEST.COM
I20260923 20:10:25.258813 12552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:10:25.244910 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48318 (local address 127.1.170.131:34829)
0923 20:10:25.245109 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0923 20:10:25.245117 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 20:10:25.245205 (+ 88us) server_negotiation.cc:407] Connection header received
0923 20:10:25.245449 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:10:25.245462 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:10:25.245532 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:10:25.245640 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:10:25.246925 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:10:25.247618 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:10:25.248925 (+ 1307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:10:25.249316 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:10:25.249448 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:10:25.253728 (+ 4280us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:10:25.253747 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:10:25.253765 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:10:25.253798 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:10:25.256442 (+ 2644us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:10:25.257073 (+ 631us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:10:25.257080 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:10:25.257082 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:10:25.257208 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:10:25.257550 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:10:25.257553 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:10:25.257555 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:10:25.257863 (+ 308us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:10:25.258132 (+ 269us) server_negotiation.cc:1035] Waiting for connection context
0923 20:10:25.258406 (+ 274us) server_negotiation.cc:299] Negotiation successful
0923 20:10:25.258558 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260923 20:10:25.259436 12531 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:10:25.348310 1706 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqUuku6/build/asan/bin/kudu with pid 12408
I20260923 20:10:25.364879 1706 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskqUuku6/build/asan/bin/kudu pid 12047
I20260923 20:10:25.366052 12109 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:10:25.474973 1706 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqUuku6/build/asan/bin/kudu with pid 12047
2026-09-23T20:10:25Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (4429 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790194121577027-1706-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 20:10:25 dist-test-slave-th4g krb5kdc[12588](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 20:10:25 dist-test-slave-th4g krb5kdc[12588](info): set up 2 sockets
Sep 23 20:10:25 dist-test-slave-th4g krb5kdc[12588](info): commencing operation
krb5kdc: starting...
W20260923 20:10:26.590982 1706 mini_kdc.cc:121] Time spent starting KDC: real 1.073s user 0.003s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790194121577027-1706-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790194121577027-1706-0/krb5kdc/test-user.keytab.
Sep 23 20:10:26 dist-test-slave-th4g krb5kdc[12588](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194226, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T20:10:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T20:10:26Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.1.170.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.1.170.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.1.170.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790194121577027-1706-0/krb5kdc/rangeradmin_127.1.170.148@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.1.170.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790194121577027-1706-0/krb5kdc/rangeradmin_127.1.170.148@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.1.170.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.1.170.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.1.170.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790194121577027-1706-0/krb5kdc/rangerlookup_127.1.170.148@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.1.170.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790194121577027-1706-0/krb5kdc/rangerlookup_127.1.170.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.1.170.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.1.170.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.1.170.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790194121577027-1706-0/krb5kdc/HTTP_127.1.170.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.1.170.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790194121577027-1706-0/krb5kdc/HTTP_127.1.170.148@KRBTEST.COM.keytab.
I20260923 20:10:26.917574 1706 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790194121577027-1706-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskqUuku6/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790194121577027-1706-0/minicluster-data/postgres -l logfile start
2026-09-23 20:10:29.801 UTC [12618] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-23 20:10:29.801 UTC [12618] LOG: listening on IPv4 address "127.1.170.148", port 47523
2026-09-23 20:10:29.806 UTC [12618] LOG: listening on Unix socket "/tmp/.s.PGSQL.47523"
2026-09-23 20:10:29.831 UTC [12621] LOG: database system was shut down at 2026-09-23 20:10:28 UTC
2026-09-23 20:10:29.836 UTC [12618] LOG: database system is ready to accept connections
I20260923 20:10:30.814554 1706 mini_postgres.cc:96] Postgres bound to 47523
2026-09-23 20:10:30.834 UTC [12629] FATAL: database "slave" does not exist
127.1.170.148:47523 - accepting connections
I20260923 20:10:30.835551 1706 mini_ranger.cc:161] Starting Ranger...
I20260923 20:10:30.862432 1706 mini_ranger.cc:84] Created miniranger Postgres user
I20260923 20:10:31.410173 1706 mini_ranger.cc:87] Created ranger Postgres database
I20260923 20:10:31.410342 1706 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790194121577027-1706-0/minicluster-data/ranger-admin
2026-09-23 20:10:31,747 [I] DB FLAVOR :POSTGRES
2026-09-23 20:10:31,748 [I] --------- Verifying Ranger DB connection ---------
2026-09-23 20:10:31,748 [I] Checking connection..
2026-09-23 20:10:31,748 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskqUuku6/build/asan/bin/postgresql.jar:/tmp/dist-test-taskqUuku6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.170.148:47523/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-23 20:10:32,018 [I] Checking connection passed.
2026-09-23 20:10:32,019 [I] --------- Verifying version history table ---------
2026-09-23 20:10:32,019 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskqUuku6/build/asan/bin/postgresql.jar:/tmp/dist-test-taskqUuku6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.170.148:47523/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-23 20:10:32,311 [I] Table x_db_version_h does not exist in database ranger
2026-09-23 20:10:32,311 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskqUuku6/build/asan/bin/postgresql.jar:/tmp/dist-test-taskqUuku6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.170.148:47523/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-23 20:10:32,580 [I] Table x_db_version_h does not exist in database ranger
2026-09-23 20:10:32,581 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-23 20:10:32,581 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskqUuku6/build/asan/bin/postgresql.jar:/tmp/dist-test-taskqUuku6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.170.148:47523/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskqUuku6/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-23 20:10:32.839 UTC [12693] WARNING: there is no transaction in progress
2026-09-23 20:10:32,846 [I] create_dbversion_catalog.sql file imported successfully
2026-09-23 20:10:32,846 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskqUuku6/build/asan/bin/postgresql.jar:/tmp/dist-test-taskqUuku6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.170.148:47523/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-23 20:10:33,121 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-23 20:10:33,121 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-23 20:10:33,122 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskqUuku6/build/asan/bin/postgresql.jar:/tmp/dist-test-taskqUuku6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.170.148:47523/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-23 20:10:33,379 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskqUuku6/build/asan/bin/postgresql.jar:/tmp/dist-test-taskqUuku6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.170.148:47523/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-23 20:10:33,633 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskqUuku6/build/asan/bin/postgresql.jar:/tmp/dist-test-taskqUuku6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.170.148:47523/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-th4g.c.gcp-upstream.internal','N') ;"
2026-09-23 20:10:33,889 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskqUuku6/build/asan/bin/postgresql.jar:/tmp/dist-test-taskqUuku6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.170.148:47523/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-23 20:10:34,136 [I] Table x_portal_user does not exist in database ranger
2026-09-23 20:10:34,137 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskqUuku6/build/asan/bin/postgresql.jar:/tmp/dist-test-taskqUuku6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.170.148:47523/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-23 20:10:34,388 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-23 20:10:34,388 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskqUuku6/build/asan/bin/postgresql.jar:/tmp/dist-test-taskqUuku6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.170.148:47523/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-23 20:10:34,642 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskqUuku6/build/asan/bin/postgresql.jar:/tmp/dist-test-taskqUuku6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.170.148:47523/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-23 20:10:34,871 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-23 20:10:34,871 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskqUuku6/build/asan/bin/postgresql.jar:/tmp/dist-test-taskqUuku6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.170.148:47523/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskqUuku6/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-23 20:10:35.928 UTC [12801] WARNING: there is no transaction in progress
2026-09-23 20:10:35.946 UTC [12801] WARNING: there is no transaction in progress
2026-09-23 20:10:35.966 UTC [12801] WARNING: there is no transaction in progress
2026-09-23 20:10:35.986 UTC [12801] WARNING: there is no transaction in progress
2026-09-23 20:10:36.006 UTC [12801] WARNING: there is no transaction in progress
2026-09-23 20:10:36.026 UTC [12801] WARNING: there is no transaction in progress
2026-09-23 20:10:36.093 UTC [12801] WARNING: there is no transaction in progress
2026-09-23 20:10:36.101 UTC [12801] WARNING: there is no transaction in progress
2026-09-23 20:10:36.120 UTC [12801] WARNING: there is no transaction in progress
2026-09-23 20:10:36.133 UTC [12801] WARNING: there is no transaction in progress
2026-09-23 20:10:36.146 UTC [12801] WARNING: there is no transaction in progress
2026-09-23 20:10:36.163 UTC [12801] WARNING: there is no transaction in progress
2026-09-23 20:10:36.177 UTC [12801] WARNING: there is no transaction in progress
2026-09-23 20:10:36.186 UTC [12801] WARNING: there is no transaction in progress
2026-09-23 20:10:36.196 UTC [12801] WARNING: there is no transaction in progress
2026-09-23 20:10:36.205 UTC [12801] WARNING: there is no transaction in progress
2026-09-23 20:10:37,027 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-23 20:10:37,027 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskqUuku6/build/asan/bin/postgresql.jar:/tmp/dist-test-taskqUuku6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.170.148:47523/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-23 20:10:37,275 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-23 20:10:37,275 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskqUuku6/build/asan/bin/postgresql.jar:/tmp/dist-test-taskqUuku6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.170.148:47523/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-23 20:10:37,552 [I] Table x_portal_user already exists in database 'ranger'
2026-09-23 20:10:37,552 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskqUuku6/build/asan/bin/postgresql.jar:/tmp/dist-test-taskqUuku6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.170.148:47523/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-23 20:10:37,796 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-23 20:10:37,796 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskqUuku6/build/asan/bin/postgresql.jar:/tmp/dist-test-taskqUuku6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.170.148:47523/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-23 20:10:38,041 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskqUuku6/build/asan/bin/postgresql.jar:/tmp/dist-test-taskqUuku6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.170.148:47523/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-23 20:10:38,289 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskqUuku6/build/asan/bin/postgresql.jar:/tmp/dist-test-taskqUuku6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.170.148:47523/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-th4g.c.gcp-upstream.internal';"
2026-09-23 20:10:38,533 [I] CORE_DB_SCHEMA import status has been updated
2026-09-23 20:10:38,533 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskqUuku6/build/asan/bin/postgresql.jar:/tmp/dist-test-taskqUuku6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.170.148:47523/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-23 20:10:38,777 [I] DB_PATCHES have already been applied
I20260923 20:10:38.783397 1706 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790194121577027-1706-0/minicluster-data/ranger-admin:/tmp/dist-test-taskqUuku6/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskqUuku6/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskqUuku6/build/asan/bin/postgresql.jar:/tmp/dist-test-taskqUuku6/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260923 20:10:38.783514 1706 mini_ranger.cc:193] Using host: 127.1.170.148
I20260923 20:10:38.798621 1706 mini_ranger.cc:227] Ranger admin URL: http://127.1.170.148:44255
Sep 23, 2026 8:10:39 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 23, 2026 8:10:39 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 23, 2026 8:10:39 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790194121577027-1706-0/minicluster-data/ranger-admin/ews/webapp
Sep 23, 2026 8:10:39 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790194121577027-1706-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 23, 2026 8:10:39 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790194121577027-1706-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 23, 2026 8:10:39 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790194121577027-1706-0/minicluster-data/ranger-admin/ews/webapp].
Sep 23, 2026 8:10:39 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260923 20:10:39.936086 1706 mini_ranger.cc:160] Time spent starting Ranger: real 9.101s user 0.006s sys 0.035s
Sep 23, 2026 8:11:00 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 23, 2026 8:11:01 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.XKeyREST
Sep 23, 2026 8:11:01 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 23, 2026 8:11:01 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 23, 2026 8:11:01 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 23, 2026 8:11:01 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 23, 2026 8:11:01 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 23, 2026 8:11:01 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 23, 2026 8:11:01 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 23, 2026 8:11:01 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 23, 2026 8:11:01 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 23, 2026 8:11:01 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 23, 2026 8:11:01 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 23, 2026 8:11:01 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 23, 2026 8:11:01 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 23, 2026 8:11:01 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 23, 2026 8:11:01 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 23, 2026 8:11:02 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260923 20:11:02.642392 1706 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.1.170.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.1.170.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.1.170.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790194121577027-1706-0/krb5kdc/rangerkms_127.1.170.148@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.1.170.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790194121577027-1706-0/krb5kdc/rangerkms_127.1.170.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.1.170.148@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.1.170.148@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.1.170.148@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790194121577027-1706-0/krb5kdc/HTTP_127.1.170.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.1.170.148@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790194121577027-1706-0/krb5kdc/HTTP_127.1.170.148@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 23 20:11:02 dist-test-slave-th4g krb5kdc[12588](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194262, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260923 20:11:02.842617 1706 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260923 20:11:02.865911 1706 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260923 20:11:03.984124 1706 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260923 20:11:03.984277 1706 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790194121577027-1706-0/minicluster-data/ranger-kms
I20260923 20:11:03.984432 1706 mini_ranger_kms.cc:227] Using postgres at 127.1.170.148:47523
2026-09-23 20:11:04,423 [I] DB FLAVOR :POSTGRES
2026-09-23 20:11:04,423 [I] --------- Verifying Ranger DB connection ---------
2026-09-23 20:11:04,423 [I] Checking connection
2026-09-23 20:11:04,423 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskqUuku6/build/asan/bin/postgresql.jar:/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790194121577027-1706-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.170.148:47523/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-23 20:11:04,666 [I] connection success
2026-09-23 20:11:04,666 [I] --------- Verifying Ranger DB tables ---------
2026-09-23 20:11:04,666 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-23 20:11:04,666 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskqUuku6/build/asan/bin/postgresql.jar:/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790194121577027-1706-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.170.148:47523/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-23 20:11:04,898 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-23 20:11:04,898 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-23 20:11:04,898 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-23 20:11:04,898 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskqUuku6/build/asan/bin/postgresql.jar:/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790194121577027-1706-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.170.148:47523/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790194121577027-1706-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-23 20:11:05,168 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260923 20:11:05.727602 1706 mini_ranger_kms.cc:310] Created kms service
I20260923 20:11:05.977711 1706 mini_ranger_kms.cc:326] Created kudu user
I20260923 20:11:06.035768 1706 mini_ranger_kms.cc:343] Created rangerkms user
Sep 23 20:11:06 dist-test-slave-th4g krb5kdc[12588](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194262, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.1.170.148@KRBTEST.COM
I20260923 20:11:06.490995 1706 mini_ranger_kms.cc:384] Added ranger policy
I20260923 20:11:06.491369 1706 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790194121577027-1706-0/minicluster-data/ranger-kms:/tmp/dist-test-taskqUuku6/build/asan/bin/postgresql.jar:/tmp/dist-test-taskqUuku6/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskqUuku6/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260923 20:11:06.491421 1706 mini_ranger_kms.cc:240] Using host: 127.1.170.148
I20260923 20:11:06.512228 1706 mini_ranger_kms.cc:276] Ranger KMS PID: 12982
I20260923 20:11:06.512420 1706 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.1.170.148:33115
Sep 23, 2026 8:11:07 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 23, 2026 8:11:07 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 23, 2026 8:11:07 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790194121577027-1706-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 23, 2026 8:11:07 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790194121577027-1706-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 23, 2026 8:11:08 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790194121577027-1706-0/minicluster-data/ranger-kms/ews/webapp].
Sep 23, 2026 8:11:08 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Sep 23 20:11:08 dist-test-slave-th4g krb5kdc[12588](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790194268, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.1.170.148@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:11:08.812582 1706 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.970s user 0.114s sys 0.346s
I20260923 20:11:08.812839 1706 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260923 20:11:08.813076 1706 mini_ranger_kms.cc:401] 127.1.170.148:33115/kms/v1/keys
Sep 23 20:11:09 dist-test-slave-th4g krb5kdc[12588](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194268, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.1.170.148@KRBTEST.COM for HTTP/127.1.170.148@KRBTEST.COM
Sep 23 20:11:09 dist-test-slave-th4g krb5kdc[12588](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194268, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.1.170.148@KRBTEST.COM for HTTP/127.1.170.148@KRBTEST.COM
Sep 23 20:11:12 dist-test-slave-th4g krb5kdc[12588](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194272, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.1.170.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.170.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.170.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790194121577027-1706-0/krb5kdc/kudu_127.1.170.190.keytab.
Entry for principal kudu/127.1.170.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790194121577027-1706-0/krb5kdc/kudu_127.1.170.190.keytab.
I20260923 20:11:12.143599 1706 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqUuku6/build/asan/bin/kudu
/tmp/dist-test-taskqUuku6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790194121577027-1706-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790194121577027-1706-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790194121577027-1706-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790194121577027-1706-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.170.190:46541
--webserver_interface=127.1.170.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790194121577027-1706-0/krb5kdc/kudu_127.1.170.190.keytab
--principal=kudu/127.1.170.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.170.148:43067
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.170.190:46541
--ranger_config_path=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790194121577027-1706-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.1.170.148:33115/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790194121577027-1706-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790194121577027-1706-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790194121577027-1706-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:11:12.498426 13032 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:11:12.498709 13032 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:11:12.498795 13032 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:11:12.506420 13032 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 20:11:12.506528 13032 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:11:12.506577 13032 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:11:12.506628 13032 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 20:11:12.506716 13032 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 20:11:12.516892 13032 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.170.148:43067
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790194121577027-1706-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790194121577027-1706-0/minicluster-data/master-0/wal
--ranger_kms_url=127.1.170.148:33115/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.1.170.190:46541
--ranger_config_path=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790194121577027-1706-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790194121577027-1706-0/krb5kdc/kudu_127.1.170.190.keytab
--principal=kudu/127.1.170.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.170.190:46541
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790194121577027-1706-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.170.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790194121577027-1706-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:11:12.518275 13032 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:11:12.519552 13032 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:11:12.532478 13037 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:11:12.532403 13040 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:11:12.532441 13038 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:11:12.532763 13032 server_base.cc:1034] running on GCE node
I20260923 20:11:12.533867 13032 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:11:12.535375 13032 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:11:12.536664 13032 hybrid_clock.cc:648] HybridClock initialized: now 1790194272536548 us; error 48 us; skew 500 ppm
I20260923 20:11:12.536964 13032 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:11:12 dist-test-slave-th4g krb5kdc[12588](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194272, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.170.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:11:12.548259 13032 init.cc:377] Logged in from keytab as kudu/127.1.170.190@KRBTEST.COM (short username kudu)
I20260923 20:11:12.549780 13032 webserver.cc:460] Webserver started at http://127.1.170.190:38245/ using document root <none> and password file <none>
I20260923 20:11:12.550565 13032 fs_manager.cc:359] Metadata directory not provided
I20260923 20:11:12.550673 13032 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:11:12.550917 13032 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 23 20:11:12 dist-test-slave-th4g krb5kdc[12588](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194272, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.170.190@KRBTEST.COM for HTTP/127.1.170.148@KRBTEST.COM
I20260923 20:11:12.605805 13032 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790194121577027-1706-0/minicluster-data/master-0/data/instance:
uuid: "c18f4886aae44650bc0fad2c61749e75"
format_stamp: "Formatted at 2026-09-23 20:11:12 on dist-test-slave-th4g"
server_key: "62e699fd9c7afbc76bc2634ac30ebb45"
server_key_iv: "758a590f2767e537436bbe1654e277b7"
server_key_version: "kuduclusterkey@0"
I20260923 20:11:12.606914 13032 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790194121577027-1706-0/minicluster-data/master-0/wal/instance:
uuid: "c18f4886aae44650bc0fad2c61749e75"
format_stamp: "Formatted at 2026-09-23 20:11:12 on dist-test-slave-th4g"
server_key: "62e699fd9c7afbc76bc2634ac30ebb45"
server_key_iv: "758a590f2767e537436bbe1654e277b7"
server_key_version: "kuduclusterkey@0"
I20260923 20:11:12.614038 13032 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.006s sys 0.001s
I20260923 20:11:12.634330 13047 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:11:12.635833 13032 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260923 20:11:12.636039 13032 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790194121577027-1706-0/minicluster-data/master-0/data,/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790194121577027-1706-0/minicluster-data/master-0/wal
uuid: "c18f4886aae44650bc0fad2c61749e75"
format_stamp: "Formatted at 2026-09-23 20:11:12 on dist-test-slave-th4g"
server_key: "62e699fd9c7afbc76bc2634ac30ebb45"
server_key_iv: "758a590f2767e537436bbe1654e277b7"
server_key_version: "kuduclusterkey@0"
I20260923 20:11:12.636268 13032 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790194121577027-1706-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790194121577027-1706-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790194121577027-1706-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:11:12.663002 13032 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:11:12.667109 13032 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:11:12.667510 13032 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:11:12.688709 13032 rpc_server.cc:307] RPC server started. Bound to: 127.1.170.190:46541
I20260923 20:11:12.688804 13098 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.170.190:46541 every 8 connection(s)
I20260923 20:11:12.690128 13032 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790194121577027-1706-0/minicluster-data/master-0/data/info.pb
I20260923 20:11:12.691845 1706 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqUuku6/build/asan/bin/kudu as pid 13032
I20260923 20:11:12.692049 1706 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790194121577027-1706-0/minicluster-data/master-0/wal/instance
I20260923 20:11:12.692714 1706 external_mini_cluster.cc:1428] Setting key 48ccb3d7b650d1ed41e84960e924916f
I20260923 20:11:12.694773 13099 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:11:12.709749 13099 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c18f4886aae44650bc0fad2c61749e75: Bootstrap starting.
I20260923 20:11:12.713625 13099 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c18f4886aae44650bc0fad2c61749e75: Neither blocks nor log segments found. Creating new log.
I20260923 20:11:12.714955 13099 log.cc:824] T 00000000000000000000000000000000 P c18f4886aae44650bc0fad2c61749e75: Log is configured to *not* fsync() on all Append() calls
I20260923 20:11:12.718346 13099 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c18f4886aae44650bc0fad2c61749e75: No bootstrap required, opened a new log
Sep 23 20:11:12 dist-test-slave-th4g krb5kdc[12588](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194272, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.170.190@KRBTEST.COM
I20260923 20:11:12.729597 13102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:11:12.694921 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47880 (local address 127.1.170.190:46541)
0923 20:11:12.706445 (+ 11524us) server_negotiation.cc:206] Beginning negotiation
0923 20:11:12.706466 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0923 20:11:12.708528 (+ 2062us) server_negotiation.cc:407] Connection header received
0923 20:11:12.710451 (+ 1923us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:11:12.710473 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:11:12.710861 (+ 388us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:11:12.711262 (+ 401us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:11:12.712549 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:11:12.713868 (+ 1319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:11:12.715224 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:11:12.716319 (+ 1095us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:11:12.716518 (+ 199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:11:12.721269 (+ 4751us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:11:12.721284 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:11:12.721309 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:11:12.721351 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:11:12.725502 (+ 4151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:11:12.726251 (+ 749us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:11:12.726296 (+ 45us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:11:12.726303 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:11:12.726442 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:11:12.726836 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:11:12.726841 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:11:12.726846 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:11:12.727401 (+ 555us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:11:12.727758 (+ 357us) server_negotiation.cc:1035] Waiting for connection context
0923 20:11:12.728191 (+ 433us) server_negotiation.cc:299] Negotiation successful
0923 20:11:12.728628 (+ 437us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":11248,"thread_start_us":10981,"threads_started":1}
I20260923 20:11:12.729951 13099 raft_consensus.cc:348] T 00000000000000000000000000000000 P c18f4886aae44650bc0fad2c61749e75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c18f4886aae44650bc0fad2c61749e75" member_type: VOTER last_known_addr { host: "127.1.170.190" port: 46541 } }
I20260923 20:11:12.730361 13099 raft_consensus.cc:374] T 00000000000000000000000000000000 P c18f4886aae44650bc0fad2c61749e75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:11:12.730459 13099 raft_consensus.cc:729] T 00000000000000000000000000000000 P c18f4886aae44650bc0fad2c61749e75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c18f4886aae44650bc0fad2c61749e75, State: Initialized, Role: FOLLOWER
I20260923 20:11:12.731071 13099 consensus_queue.cc:260] T 00000000000000000000000000000000 P c18f4886aae44650bc0fad2c61749e75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c18f4886aae44650bc0fad2c61749e75" member_type: VOTER last_known_addr { host: "127.1.170.190" port: 46541 } }
I20260923 20:11:12.731351 13099 raft_consensus.cc:388] T 00000000000000000000000000000000 P c18f4886aae44650bc0fad2c61749e75 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 20:11:12.731523 13099 raft_consensus.cc:482] T 00000000000000000000000000000000 P c18f4886aae44650bc0fad2c61749e75 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 20:11:12.731688 13099 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c18f4886aae44650bc0fad2c61749e75 [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:11:12.735031 13099 raft_consensus.cc:504] T 00000000000000000000000000000000 P c18f4886aae44650bc0fad2c61749e75 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c18f4886aae44650bc0fad2c61749e75" member_type: VOTER last_known_addr { host: "127.1.170.190" port: 46541 } }
I20260923 20:11:12.735581 13099 leader_election.cc:302] T 00000000000000000000000000000000 P c18f4886aae44650bc0fad2c61749e75 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c18f4886aae44650bc0fad2c61749e75; no voters:
I20260923 20:11:12.736219 13099 leader_election.cc:288] T 00000000000000000000000000000000 P c18f4886aae44650bc0fad2c61749e75 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 20:11:12.736382 13104 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c18f4886aae44650bc0fad2c61749e75 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 20:11:12.738910 13099 sys_catalog.cc:564] T 00000000000000000000000000000000 P c18f4886aae44650bc0fad2c61749e75 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 20:11:12.739480 13104 raft_consensus.cc:686] T 00000000000000000000000000000000 P c18f4886aae44650bc0fad2c61749e75 [term 1 LEADER]: Becoming Leader. State: Replica: c18f4886aae44650bc0fad2c61749e75, State: Running, Role: LEADER
I20260923 20:11:12.740125 13104 consensus_queue.cc:237] T 00000000000000000000000000000000 P c18f4886aae44650bc0fad2c61749e75 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c18f4886aae44650bc0fad2c61749e75" member_type: VOTER last_known_addr { host: "127.1.170.190" port: 46541 } }
I20260923 20:11:12.743363 13105 sys_catalog.cc:455] T 00000000000000000000000000000000 P c18f4886aae44650bc0fad2c61749e75 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c18f4886aae44650bc0fad2c61749e75" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c18f4886aae44650bc0fad2c61749e75" member_type: VOTER last_known_addr { host: "127.1.170.190" port: 46541 } } }
I20260923 20:11:12.743813 13105 sys_catalog.cc:458] T 00000000000000000000000000000000 P c18f4886aae44650bc0fad2c61749e75 [sys.catalog]: This master's current role is: LEADER
I20260923 20:11:12.746769 13106 sys_catalog.cc:455] T 00000000000000000000000000000000 P c18f4886aae44650bc0fad2c61749e75 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c18f4886aae44650bc0fad2c61749e75. Latest consensus state: current_term: 1 leader_uuid: "c18f4886aae44650bc0fad2c61749e75" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c18f4886aae44650bc0fad2c61749e75" member_type: VOTER last_known_addr { host: "127.1.170.190" port: 46541 } } }
I20260923 20:11:12.747215 13106 sys_catalog.cc:458] T 00000000000000000000000000000000 P c18f4886aae44650bc0fad2c61749e75 [sys.catalog]: This master's current role is: LEADER
I20260923 20:11:12.747797 13111 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 20:11:12.757642 13111 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 20:11:12.783857 13111 catalog_manager.cc:1347] Generated new cluster ID: 645aaa8370404959a2e595e77478bc97
I20260923 20:11:12.783980 13111 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 20:11:12.800385 13111 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 20:11:12.802366 13111 catalog_manager.cc:1504] Loading token signing keys...
I20260923 20:11:12.814653 13099 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790194121577027-1706-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260923 20:11:12.815327 13111 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c18f4886aae44650bc0fad2c61749e75: Generated new TSK 0
I20260923 20:11:12.816466 13111 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.170.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.170.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.170.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790194121577027-1706-0/krb5kdc/kudu_127.1.170.129.keytab.
Entry for principal kudu/127.1.170.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790194121577027-1706-0/krb5kdc/kudu_127.1.170.129.keytab.
I20260923 20:11:13.594136 1706 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqUuku6/build/asan/bin/kudu
/tmp/dist-test-taskqUuku6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790194121577027-1706-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790194121577027-1706-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790194121577027-1706-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790194121577027-1706-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.1.170.148:33115/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.170.129:0
--local_ip_for_outbound_sockets=127.1.170.129
--webserver_interface=127.1.170.129
--webserver_port=0
--tserver_master_addrs=127.1.170.190:46541
--keytab_file=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790194121577027-1706-0/krb5kdc/kudu_127.1.170.129.keytab
--principal=kudu/127.1.170.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.170.148:43067
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790194121577027-1706-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790194121577027-1706-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790194121577027-1706-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 23 20:11:13 dist-test-slave-th4g krb5kdc[12588](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790194273, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.170.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260923 20:11:14.023587 13145 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:11:14.024000 13145 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:11:14.024205 13145 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:11:14.033783 13145 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:11:14.034011 13145 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:11:14.034209 13145 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.170.129
I20260923 20:11:14.050325 13145 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.170.148:43067
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790194121577027-1706-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790194121577027-1706-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.1.170.148:33115/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790194121577027-1706-0/krb5kdc/kudu_127.1.170.129.keytab
--principal=kudu/127.1.170.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.170.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790194121577027-1706-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.170.129
--webserver_port=0
--tserver_master_addrs=127.1.170.190:46541
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.170.129
--log_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790194121577027-1706-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:11:14.052245 13145 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:11:14.054091 13145 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:11:14.073004 13155 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:11:14.073791 13156 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:11:14.091778 13158 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:11:14.092180 13145 server_base.cc:1034] running on GCE node
I20260923 20:11:14.092931 13145 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:11:14.094091 13145 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:11:14.095382 13145 hybrid_clock.cc:648] HybridClock initialized: now 1790194274095316 us; error 59 us; skew 500 ppm
I20260923 20:11:14.095732 13145 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:11:14 dist-test-slave-th4g krb5kdc[12588](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194274, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.170.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:11:14.099407 13145 init.cc:377] Logged in from keytab as kudu/127.1.170.129@KRBTEST.COM (short username kudu)
I20260923 20:11:14.101312 13145 webserver.cc:460] Webserver started at http://127.1.170.129:39543/ using document root <none> and password file <none>
I20260923 20:11:14.102276 13145 fs_manager.cc:359] Metadata directory not provided
I20260923 20:11:14.102463 13145 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:11:14.102813 13145 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 23 20:11:14 dist-test-slave-th4g krb5kdc[12588](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194274, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.170.129@KRBTEST.COM for HTTP/127.1.170.148@KRBTEST.COM
I20260923 20:11:14.132093 13145 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790194121577027-1706-0/minicluster-data/ts-0/data/instance:
uuid: "1a0e5edef160421183713f47489fe44e"
format_stamp: "Formatted at 2026-09-23 20:11:14 on dist-test-slave-th4g"
server_key: "fb20824b869a43f1ec1fa7486a4ab1e9"
server_key_iv: "bc86f5cea08200afa610a9e2f74cf877"
server_key_version: "kuduclusterkey@0"
I20260923 20:11:14.133347 13145 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790194121577027-1706-0/minicluster-data/ts-0/wal/instance:
uuid: "1a0e5edef160421183713f47489fe44e"
format_stamp: "Formatted at 2026-09-23 20:11:14 on dist-test-slave-th4g"
server_key: "fb20824b869a43f1ec1fa7486a4ab1e9"
server_key_iv: "bc86f5cea08200afa610a9e2f74cf877"
server_key_version: "kuduclusterkey@0"
I20260923 20:11:14.152678 13145 fs_manager.cc:691] Time spent creating directory manager: real 0.019s user 0.007s sys 0.011s
I20260923 20:11:14.177381 13165 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:11:14.179519 13145 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.001s
I20260923 20:11:14.179760 13145 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790194121577027-1706-0/minicluster-data/ts-0/data,/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790194121577027-1706-0/minicluster-data/ts-0/wal
uuid: "1a0e5edef160421183713f47489fe44e"
format_stamp: "Formatted at 2026-09-23 20:11:14 on dist-test-slave-th4g"
server_key: "fb20824b869a43f1ec1fa7486a4ab1e9"
server_key_iv: "bc86f5cea08200afa610a9e2f74cf877"
server_key_version: "kuduclusterkey@0"
I20260923 20:11:14.180071 13145 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790194121577027-1706-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790194121577027-1706-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790194121577027-1706-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:11:14.227442 13145 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:11:14.233163 13145 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:11:14.233589 13145 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:11:14.241719 13145 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 20:11:14.250675 13145 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 20:11:14.250871 13145 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260923 20:11:14.251080 13145 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 20:11:14.251194 13145 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 23 20:11:14 dist-test-slave-th4g krb5kdc[12588](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194273, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.170.190@KRBTEST.COM for HTTP/127.1.170.148@KRBTEST.COM
I20260923 20:11:14.546139 13145 rpc_server.cc:307] RPC server started. Bound to: 127.1.170.129:43565
I20260923 20:11:14.546161 13277 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.170.129:43565 every 8 connection(s)
I20260923 20:11:14.548012 13145 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790194121577027-1706-0/minicluster-data/ts-0/data/info.pb
I20260923 20:11:14.555864 1706 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqUuku6/build/asan/bin/kudu as pid 13145
I20260923 20:11:14.556064 1706 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790194121577027-1706-0/minicluster-data/ts-0/wal/instance
I20260923 20:11:14.556869 1706 external_mini_cluster.cc:1428] Setting key d10aa861acb069dbc6358d6240609bc3
Sep 23 20:11:14 dist-test-slave-th4g krb5kdc[12588](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194274, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.170.129@KRBTEST.COM for kudu/127.1.170.190@KRBTEST.COM
I20260923 20:11:14.580731 13281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:11:14.553016 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.170.129:53755 (local address 127.1.170.190:46541)
0923 20:11:14.559041 (+ 6025us) server_negotiation.cc:206] Beginning negotiation
0923 20:11:14.559048 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:11:14.559066 (+ 18us) server_negotiation.cc:407] Connection header received
0923 20:11:14.559118 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:11:14.559124 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:11:14.559209 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:11:14.559302 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:11:14.562039 (+ 2737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:11:14.563223 (+ 1184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:11:14.566442 (+ 3219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:11:14.567541 (+ 1099us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:11:14.567664 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:11:14.572554 (+ 4890us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:11:14.572562 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:11:14.572567 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:11:14.572618 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:11:14.576426 (+ 3808us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:11:14.577594 (+ 1168us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:11:14.577600 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:11:14.577603 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:11:14.577765 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:11:14.578405 (+ 640us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:11:14.578410 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:11:14.578413 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:11:14.578773 (+ 360us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:11:14.578909 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0923 20:11:14.579969 (+ 1060us) server_negotiation.cc:299] Negotiation successful
0923 20:11:14.580220 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5868,"thread_start_us":5761,"threads_started":1}
I20260923 20:11:14.583458 13280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:11:14.552339 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.190:46541 (local address 127.1.170.129:53755)
0923 20:11:14.555592 (+ 3253us) negotiation.cc:107] Waiting for socket to connect
0923 20:11:14.555657 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0923 20:11:14.558298 (+ 2641us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:11:14.560321 (+ 2023us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:11:14.560333 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:11:14.560958 (+ 625us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:11:14.561758 (+ 800us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:11:14.561784 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:11:14.563496 (+ 1712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:11:14.563502 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:11:14.566232 (+ 2730us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:11:14.566273 (+ 41us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:11:14.568026 (+ 1753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:11:14.568034 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:11:14.568349 (+ 315us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:11:14.569182 (+ 833us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:11:14.569212 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:11:14.572229 (+ 3017us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:11:14.576688 (+ 4459us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:11:14.576698 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:11:14.576720 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:11:14.577427 (+ 707us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:11:14.577945 (+ 518us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:11:14.577949 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:11:14.577953 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:11:14.578265 (+ 312us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:11:14.579005 (+ 740us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:11:14.579016 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:11:14.579506 (+ 490us) client_negotiation.cc:769] Sending connection context
0923 20:11:14.581410 (+ 1904us) client_negotiation.cc:240] Negotiation successful
0923 20:11:14.582006 (+ 596us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2883,"thread_start_us":2636,"threads_started":1}
I20260923 20:11:14.585997 13278 heartbeater.cc:347] Connected to a master server at 127.1.170.190:46541
I20260923 20:11:14.586355 13278 heartbeater.cc:464] Registering TS with master...
I20260923 20:11:14.587301 13278 heartbeater.cc:511] Master 127.1.170.190:46541 requested a full tablet report, sending...
I20260923 20:11:14.590181 13064 ts_manager.cc:194] Registered new tserver with Master: 1a0e5edef160421183713f47489fe44e (127.1.170.129:43565)
I20260923 20:11:14.592595 13064 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.170.129@KRBTEST.COM'} at 127.1.170.129:53755
WARNING: no policy specified for kudu/127.1.170.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.170.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.170.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790194121577027-1706-0/krb5kdc/kudu_127.1.170.130.keytab.
Entry for principal kudu/127.1.170.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790194121577027-1706-0/krb5kdc/kudu_127.1.170.130.keytab.
I20260923 20:11:14.638870 1706 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqUuku6/build/asan/bin/kudu
/tmp/dist-test-taskqUuku6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790194121577027-1706-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790194121577027-1706-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790194121577027-1706-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790194121577027-1706-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.1.170.148:33115/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.170.130:0
--local_ip_for_outbound_sockets=127.1.170.130
--webserver_interface=127.1.170.130
--webserver_port=0
--tserver_master_addrs=127.1.170.190:46541
--keytab_file=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790194121577027-1706-0/krb5kdc/kudu_127.1.170.130.keytab
--principal=kudu/127.1.170.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.170.148:43067
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790194121577027-1706-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790194121577027-1706-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790194121577027-1706-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 23 20:11:14 dist-test-slave-th4g krb5kdc[12588](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194273, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.170.190@KRBTEST.COM for HTTP/127.1.170.148@KRBTEST.COM
W20260923 20:11:15.119580 13284 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:11:15.119874 13284 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:11:15.120023 13284 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:11:15.127668 13284 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:11:15.127790 13284 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:11:15.127969 13284 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.170.130
I20260923 20:11:15.139559 13284 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.170.148:43067
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790194121577027-1706-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790194121577027-1706-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.1.170.148:33115/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790194121577027-1706-0/krb5kdc/kudu_127.1.170.130.keytab
--principal=kudu/127.1.170.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.170.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790194121577027-1706-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.170.130
--webserver_port=0
--tserver_master_addrs=127.1.170.190:46541
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.170.130
--log_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790194121577027-1706-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:11:15.141098 13284 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:11:15.142482 13284 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:11:15.158067 13299 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:11:15.158186 13301 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:11:15.158213 13298 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:11:15.160275 13284 server_base.cc:1034] running on GCE node
I20260923 20:11:15.161065 13284 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:11:15.162284 13284 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:11:15.163544 13284 hybrid_clock.cc:648] HybridClock initialized: now 1790194275163453 us; error 67 us; skew 500 ppm
I20260923 20:11:15.163908 13284 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:11:15 dist-test-slave-th4g krb5kdc[12588](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194275, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.170.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:11:15.168658 13284 init.cc:377] Logged in from keytab as kudu/127.1.170.130@KRBTEST.COM (short username kudu)
I20260923 20:11:15.170310 13284 webserver.cc:460] Webserver started at http://127.1.170.130:36823/ using document root <none> and password file <none>
I20260923 20:11:15.171021 13284 fs_manager.cc:359] Metadata directory not provided
I20260923 20:11:15.171147 13284 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:11:15.171463 13284 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 23 20:11:15 dist-test-slave-th4g krb5kdc[12588](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194275, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.170.130@KRBTEST.COM for HTTP/127.1.170.148@KRBTEST.COM
I20260923 20:11:15.190784 13284 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790194121577027-1706-0/minicluster-data/ts-1/data/instance:
uuid: "9409360d42ca4922b1c09ee4c93511a4"
format_stamp: "Formatted at 2026-09-23 20:11:15 on dist-test-slave-th4g"
server_key: "8e8e6e200cbe19231cc7fcc7bddaf472"
server_key_iv: "43e181bea0a6cc85c599fff04cdc7417"
server_key_version: "kuduclusterkey@0"
I20260923 20:11:15.192562 13284 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790194121577027-1706-0/minicluster-data/ts-1/wal/instance:
uuid: "9409360d42ca4922b1c09ee4c93511a4"
format_stamp: "Formatted at 2026-09-23 20:11:15 on dist-test-slave-th4g"
server_key: "8e8e6e200cbe19231cc7fcc7bddaf472"
server_key_iv: "43e181bea0a6cc85c599fff04cdc7417"
server_key_version: "kuduclusterkey@0"
I20260923 20:11:15.199848 13284 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.006s sys 0.000s
I20260923 20:11:15.217818 13308 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:11:15.219736 13284 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.002s
I20260923 20:11:15.219938 13284 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790194121577027-1706-0/minicluster-data/ts-1/data,/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790194121577027-1706-0/minicluster-data/ts-1/wal
uuid: "9409360d42ca4922b1c09ee4c93511a4"
format_stamp: "Formatted at 2026-09-23 20:11:15 on dist-test-slave-th4g"
server_key: "8e8e6e200cbe19231cc7fcc7bddaf472"
server_key_iv: "43e181bea0a6cc85c599fff04cdc7417"
server_key_version: "kuduclusterkey@0"
I20260923 20:11:15.220214 13284 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790194121577027-1706-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790194121577027-1706-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790194121577027-1706-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:11:15.257570 13284 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:11:15.261926 13284 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:11:15.262255 13284 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:11:15.263386 13284 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 20:11:15.264985 13284 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 20:11:15.265061 13284 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 20:11:15.265208 13284 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 20:11:15.265277 13284 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 20:11:15.306175 13284 rpc_server.cc:307] RPC server started. Bound to: 127.1.170.130:36713
I20260923 20:11:15.306175 13420 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.170.130:36713 every 8 connection(s)
I20260923 20:11:15.307907 13284 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790194121577027-1706-0/minicluster-data/ts-1/data/info.pb
I20260923 20:11:15.312691 1706 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqUuku6/build/asan/bin/kudu as pid 13284
I20260923 20:11:15.312906 1706 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790194121577027-1706-0/minicluster-data/ts-1/wal/instance
I20260923 20:11:15.313699 1706 external_mini_cluster.cc:1428] Setting key a4a4440a2694330936edd6ed97f0de58
Sep 23 20:11:15 dist-test-slave-th4g krb5kdc[12588](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194275, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.170.130@KRBTEST.COM for kudu/127.1.170.190@KRBTEST.COM
I20260923 20:11:15.335158 13423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:11:15.311366 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.170.130:51889 (local address 127.1.170.190:46541)
0923 20:11:15.311903 (+ 537us) server_negotiation.cc:206] Beginning negotiation
0923 20:11:15.311910 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:11:15.317985 (+ 6075us) server_negotiation.cc:407] Connection header received
0923 20:11:15.318086 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:11:15.318090 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:11:15.318173 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:11:15.318313 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:11:15.319867 (+ 1554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:11:15.320607 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:11:15.322531 (+ 1924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:11:15.322956 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:11:15.323098 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:11:15.327124 (+ 4026us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:11:15.327131 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:11:15.327136 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:11:15.327183 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:11:15.330886 (+ 3703us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:11:15.331940 (+ 1054us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:11:15.331946 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:11:15.331949 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:11:15.332096 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:11:15.332634 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:11:15.332646 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:11:15.332649 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:11:15.333110 (+ 461us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:11:15.333240 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0923 20:11:15.334587 (+ 1347us) server_negotiation.cc:299] Negotiation successful
0923 20:11:15.334811 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":405,"thread_start_us":271,"threads_started":1}
I20260923 20:11:15.336198 13424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:11:15.311927 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.190:46541 (local address 127.1.170.130:51889)
0923 20:11:15.315091 (+ 3164us) negotiation.cc:107] Waiting for socket to connect
0923 20:11:15.315154 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0923 20:11:15.316940 (+ 1786us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:11:15.318599 (+ 1659us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:11:15.318608 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:11:15.319076 (+ 468us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:11:15.319685 (+ 609us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:11:15.319702 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:11:15.320773 (+ 1071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:11:15.320778 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:11:15.322374 (+ 1596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:11:15.322388 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:11:15.323224 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:11:15.323231 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:11:15.323471 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:11:15.324192 (+ 721us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:11:15.324216 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:11:15.326795 (+ 2579us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:11:15.331087 (+ 4292us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:11:15.331101 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:11:15.331124 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:11:15.331782 (+ 658us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:11:15.332234 (+ 452us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:11:15.332241 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:11:15.332248 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:11:15.332499 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:11:15.333226 (+ 727us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:11:15.333237 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:11:15.333716 (+ 479us) client_negotiation.cc:769] Sending connection context
0923 20:11:15.334073 (+ 357us) client_negotiation.cc:240] Negotiation successful
0923 20:11:15.334672 (+ 599us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2878,"thread_start_us":2664,"threads_started":1}
I20260923 20:11:15.337922 13421 heartbeater.cc:347] Connected to a master server at 127.1.170.190:46541
I20260923 20:11:15.338265 13421 heartbeater.cc:464] Registering TS with master...
I20260923 20:11:15.339170 13421 heartbeater.cc:511] Master 127.1.170.190:46541 requested a full tablet report, sending...
I20260923 20:11:15.341202 13064 ts_manager.cc:194] Registered new tserver with Master: 9409360d42ca4922b1c09ee4c93511a4 (127.1.170.130:36713)
WARNING: no policy specified for kudu/127.1.170.131@KRBTEST.COM; defaulting to no policy
I20260923 20:11:15.342247 13064 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.170.130@KRBTEST.COM'} at 127.1.170.130:51889
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.170.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.170.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790194121577027-1706-0/krb5kdc/kudu_127.1.170.131.keytab.
Entry for principal kudu/127.1.170.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790194121577027-1706-0/krb5kdc/kudu_127.1.170.131.keytab.
I20260923 20:11:15.374315 1706 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqUuku6/build/asan/bin/kudu
/tmp/dist-test-taskqUuku6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790194121577027-1706-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790194121577027-1706-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790194121577027-1706-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790194121577027-1706-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.1.170.148:33115/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.170.131:0
--local_ip_for_outbound_sockets=127.1.170.131
--webserver_interface=127.1.170.131
--webserver_port=0
--tserver_master_addrs=127.1.170.190:46541
--keytab_file=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790194121577027-1706-0/krb5kdc/kudu_127.1.170.131.keytab
--principal=kudu/127.1.170.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.170.148:43067
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790194121577027-1706-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790194121577027-1706-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790194121577027-1706-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260923 20:11:15.596908 13278 heartbeater.cc:503] Master 127.1.170.190:46541 was elected leader, sending a full tablet report...
W20260923 20:11:15.740985 13427 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:11:15.741314 13427 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:11:15.741499 13427 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:11:15.749184 13427 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:11:15.749306 13427 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:11:15.749490 13427 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.170.131
I20260923 20:11:15.761044 13427 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.170.148:43067
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790194121577027-1706-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790194121577027-1706-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.1.170.148:33115/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790194121577027-1706-0/krb5kdc/kudu_127.1.170.131.keytab
--principal=kudu/127.1.170.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.170.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790194121577027-1706-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.170.131
--webserver_port=0
--tserver_master_addrs=127.1.170.190:46541
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.170.131
--log_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790194121577027-1706-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:11:15.762949 13427 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:11:15.764760 13427 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:11:15.780055 13433 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:11:15.780109 13435 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:11:15.780158 13432 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:11:15.783799 13427 server_base.cc:1034] running on GCE node
I20260923 20:11:15.784489 13427 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:11:15.785629 13427 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:11:15.787151 13427 hybrid_clock.cc:648] HybridClock initialized: now 1790194275787092 us; error 46 us; skew 500 ppm
I20260923 20:11:15.787578 13427 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:11:15 dist-test-slave-th4g krb5kdc[12588](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194275, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.170.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:11:15.800500 13427 init.cc:377] Logged in from keytab as kudu/127.1.170.131@KRBTEST.COM (short username kudu)
I20260923 20:11:15.802160 13427 webserver.cc:460] Webserver started at http://127.1.170.131:38789/ using document root <none> and password file <none>
I20260923 20:11:15.802886 13427 fs_manager.cc:359] Metadata directory not provided
I20260923 20:11:15.803036 13427 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:11:15.803318 13427 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 23 20:11:15 dist-test-slave-th4g krb5kdc[12588](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194275, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.170.131@KRBTEST.COM for HTTP/127.1.170.148@KRBTEST.COM
I20260923 20:11:15.820564 13427 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790194121577027-1706-0/minicluster-data/ts-2/data/instance:
uuid: "54637695ec69409795bd3fba64c7a416"
format_stamp: "Formatted at 2026-09-23 20:11:15 on dist-test-slave-th4g"
server_key: "7bd4515c4075cc364a99e5d8947f9882"
server_key_iv: "c08b6fcd3c9ce1a7717c00f343983e3b"
server_key_version: "kuduclusterkey@0"
I20260923 20:11:15.821740 13427 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790194121577027-1706-0/minicluster-data/ts-2/wal/instance:
uuid: "54637695ec69409795bd3fba64c7a416"
format_stamp: "Formatted at 2026-09-23 20:11:15 on dist-test-slave-th4g"
server_key: "7bd4515c4075cc364a99e5d8947f9882"
server_key_iv: "c08b6fcd3c9ce1a7717c00f343983e3b"
server_key_version: "kuduclusterkey@0"
I20260923 20:11:15.828778 13427 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.001s sys 0.004s
I20260923 20:11:15.847412 13442 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:11:15.849275 13427 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260923 20:11:15.849524 13427 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790194121577027-1706-0/minicluster-data/ts-2/data,/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790194121577027-1706-0/minicluster-data/ts-2/wal
uuid: "54637695ec69409795bd3fba64c7a416"
format_stamp: "Formatted at 2026-09-23 20:11:15 on dist-test-slave-th4g"
server_key: "7bd4515c4075cc364a99e5d8947f9882"
server_key_iv: "c08b6fcd3c9ce1a7717c00f343983e3b"
server_key_version: "kuduclusterkey@0"
I20260923 20:11:15.849923 13427 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790194121577027-1706-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790194121577027-1706-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790194121577027-1706-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:11:15.881531 13427 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:11:15.887308 13427 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:11:15.887647 13427 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:11:15.889277 13427 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 20:11:15.891147 13427 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 20:11:15.891294 13427 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260923 20:11:15.891443 13427 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 20:11:15.891505 13427 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 20:11:15.959568 13427 rpc_server.cc:307] RPC server started. Bound to: 127.1.170.131:33467
I20260923 20:11:15.959663 13554 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.170.131:33467 every 8 connection(s)
I20260923 20:11:15.960855 13427 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790194121577027-1706-0/minicluster-data/ts-2/data/info.pb
I20260923 20:11:15.963965 1706 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqUuku6/build/asan/bin/kudu as pid 13427
I20260923 20:11:15.964126 1706 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790194121577027-1706-0/minicluster-data/ts-2/wal/instance
I20260923 20:11:15.964655 1706 external_mini_cluster.cc:1428] Setting key 51fe7b766a5fe61c60b3cff2be55b2a8
Sep 23 20:11:15 dist-test-slave-th4g krb5kdc[12588](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194275, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.170.131@KRBTEST.COM for kudu/127.1.170.190@KRBTEST.COM
I20260923 20:11:15.994586 13557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:11:15.964153 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.170.131:40985 (local address 127.1.170.190:46541)
0923 20:11:15.966147 (+ 1994us) server_negotiation.cc:206] Beginning negotiation
0923 20:11:15.966156 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 20:11:15.970673 (+ 4517us) server_negotiation.cc:407] Connection header received
0923 20:11:15.973224 (+ 2551us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:11:15.973229 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:11:15.973308 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:11:15.973414 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:11:15.976611 (+ 3197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:11:15.977771 (+ 1160us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:11:15.980510 (+ 2739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:11:15.981276 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:11:15.981432 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:11:15.986427 (+ 4995us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:11:15.986435 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:11:15.986441 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:11:15.986495 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:11:15.990675 (+ 4180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:11:15.991702 (+ 1027us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:11:15.991714 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:11:15.991718 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:11:15.991886 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:11:15.992630 (+ 744us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:11:15.992636 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:11:15.992639 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:11:15.992962 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:11:15.993146 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0923 20:11:15.994031 (+ 885us) server_negotiation.cc:299] Negotiation successful
0923 20:11:15.994211 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1866,"thread_start_us":1708,"threads_started":1}
I20260923 20:11:15.995915 13558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:11:15.965245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.190:46541 (local address 127.1.170.131:40985)
0923 20:11:15.970499 (+ 5254us) negotiation.cc:107] Waiting for socket to connect
0923 20:11:15.970550 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0923 20:11:15.972901 (+ 2351us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:11:15.973675 (+ 774us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:11:15.973698 (+ 23us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:11:15.974696 (+ 998us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:11:15.976364 (+ 1668us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:11:15.976393 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:11:15.978185 (+ 1792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:11:15.978191 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:11:15.980268 (+ 2077us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:11:15.980295 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:11:15.981531 (+ 1236us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:11:15.981542 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:11:15.981818 (+ 276us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:11:15.982645 (+ 827us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:11:15.982669 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:11:15.986050 (+ 3381us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:11:15.990919 (+ 4869us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:11:15.990929 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:11:15.990947 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:11:15.991442 (+ 495us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:11:15.992022 (+ 580us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:11:15.992028 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:11:15.992032 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:11:15.992350 (+ 318us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:11:15.993119 (+ 769us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:11:15.993128 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:11:15.993584 (+ 456us) client_negotiation.cc:769] Sending connection context
0923 20:11:15.993940 (+ 356us) client_negotiation.cc:240] Negotiation successful
0923 20:11:15.994507 (+ 567us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4912,"thread_start_us":4633,"threads_started":1}
I20260923 20:11:15.997501 13555 heartbeater.cc:347] Connected to a master server at 127.1.170.190:46541
I20260923 20:11:15.997848 13555 heartbeater.cc:464] Registering TS with master...
I20260923 20:11:15.998715 13555 heartbeater.cc:511] Master 127.1.170.190:46541 requested a full tablet report, sending...
I20260923 20:11:16.000579 13064 ts_manager.cc:194] Registered new tserver with Master: 54637695ec69409795bd3fba64c7a416 (127.1.170.131:33467)
I20260923 20:11:16.001451 13064 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.170.131@KRBTEST.COM'} at 127.1.170.131:40985
I20260923 20:11:16.006700 1706 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260923 20:11:16.028021 13557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:11:16.012420 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47708 (local address 127.1.170.190:46541)
0923 20:11:16.012691 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0923 20:11:16.012701 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0923 20:11:16.013191 (+ 490us) server_negotiation.cc:407] Connection header received
0923 20:11:16.013393 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:11:16.013397 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:11:16.013477 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:11:16.013579 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:11:16.015590 (+ 2011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:11:16.016361 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:11:16.018391 (+ 2030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:11:16.018829 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:11:16.018935 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:11:16.022058 (+ 3123us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:11:16.022063 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:11:16.022066 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:11:16.022098 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:11:16.024939 (+ 2841us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:11:16.025985 (+ 1046us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:11:16.025990 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:11:16.025992 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:11:16.026113 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:11:16.026528 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:11:16.026540 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:11:16.026542 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:11:16.026839 (+ 297us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:11:16.027083 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0923 20:11:16.027418 (+ 335us) server_negotiation.cc:299] Negotiation successful
0923 20:11:16.027639 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260923 20:11:16.032317 13064 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47708:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 20:11:16.040165 13064 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 20:11:16.072732 13568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:11:16.053571 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.129:43565 (local address 127.0.0.1:37916)
0923 20:11:16.055442 (+ 1871us) negotiation.cc:107] Waiting for socket to connect
0923 20:11:16.055478 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0923 20:11:16.055657 (+ 179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:11:16.057953 (+ 2296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:11:16.057963 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:11:16.058000 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:11:16.058655 (+ 655us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:11:16.058670 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:11:16.062816 (+ 4146us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:11:16.062824 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:11:16.067047 (+ 4223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:11:16.067067 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:11:16.071496 (+ 4429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:11:16.071505 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:11:16.071787 (+ 282us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:11:16.071855 (+ 68us) client_negotiation.cc:769] Sending connection context
0923 20:11:16.071993 (+ 138us) client_negotiation.cc:240] Negotiation successful
0923 20:11:16.072246 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1635,"thread_start_us":1631,"threads_started":1}
I20260923 20:11:16.074120 13570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:11:16.054088 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37916 (local address 127.1.170.129:43565)
0923 20:11:16.054907 (+ 819us) server_negotiation.cc:206] Beginning negotiation
0923 20:11:16.054919 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0923 20:11:16.055833 (+ 914us) server_negotiation.cc:407] Connection header received
0923 20:11:16.055935 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:11:16.055945 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:11:16.056200 (+ 255us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:11:16.057607 (+ 1407us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:11:16.058891 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:11:16.062629 (+ 3738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:11:16.067278 (+ 4649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:11:16.070889 (+ 3611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:11:16.071057 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:11:16.071357 (+ 300us) server_negotiation.cc:1035] Waiting for connection context
0923 20:11:16.073316 (+ 1959us) server_negotiation.cc:299] Negotiation successful
0923 20:11:16.073569 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":637,"thread_start_us":952,"threads_started":1}
I20260923 20:11:16.078740 13213 tablet_service.cc:1467] Processing CreateTablet for tablet fcbe9ee8283e4dc3a8f5e461a96956ed (DEFAULT_TABLE table=test-table [id=4976ec40ae874cfa9aa672de95ca0cb4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 20:11:16.080538 13213 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fcbe9ee8283e4dc3a8f5e461a96956ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:11:16.082494 13568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:11:16.056007 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.130:36713 (local address 127.0.0.1:48020)
0923 20:11:16.072977 (+ 16970us) negotiation.cc:107] Waiting for socket to connect
0923 20:11:16.072996 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0923 20:11:16.073116 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:11:16.075298 (+ 2182us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:11:16.075303 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:11:16.075328 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:11:16.075517 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:11:16.075528 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:11:16.077413 (+ 1885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:11:16.077418 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:11:16.079303 (+ 1885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:11:16.079317 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:11:16.081715 (+ 2398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:11:16.081721 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:11:16.081924 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:11:16.081957 (+ 33us) client_negotiation.cc:769] Sending connection context
0923 20:11:16.082045 (+ 88us) client_negotiation.cc:240] Negotiation successful
0923 20:11:16.082203 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16882,"thread_start_us":18111,"threads_started":1}
I20260923 20:11:16.083233 13573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:11:16.056334 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48020 (local address 127.1.170.130:36713)
0923 20:11:16.074784 (+ 18450us) server_negotiation.cc:206] Beginning negotiation
0923 20:11:16.074795 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0923 20:11:16.074828 (+ 33us) server_negotiation.cc:407] Connection header received
0923 20:11:16.074904 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:11:16.074910 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:11:16.075103 (+ 193us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:11:16.075317 (+ 214us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:11:16.075675 (+ 358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:11:16.077243 (+ 1568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:11:16.079492 (+ 2249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:11:16.081306 (+ 1814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:11:16.081443 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:11:16.081623 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0923 20:11:16.082718 (+ 1095us) server_negotiation.cc:299] Negotiation successful
0923 20:11:16.082878 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18289,"thread_start_us":18172,"threads_started":1}
I20260923 20:11:16.084882 13569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:11:16.054080 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.131:33467 (local address 127.0.0.1:56224)
0923 20:11:16.065198 (+ 11118us) negotiation.cc:107] Waiting for socket to connect
0923 20:11:16.065225 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0923 20:11:16.065365 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:11:16.076564 (+ 11199us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:11:16.076571 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:11:16.076601 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:11:16.076984 (+ 383us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:11:16.076995 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:11:16.079053 (+ 2058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:11:16.079058 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:11:16.081109 (+ 2051us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:11:16.081124 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:11:16.084018 (+ 2894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:11:16.084024 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:11:16.084209 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:11:16.084241 (+ 32us) client_negotiation.cc:769] Sending connection context
0923 20:11:16.084332 (+ 91us) client_negotiation.cc:240] Negotiation successful
0923 20:11:16.084465 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10881,"thread_start_us":11395,"threads_started":1}
I20260923 20:11:16.085726 13571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:11:16.054734 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56224 (local address 127.1.170.131:33467)
0923 20:11:16.074857 (+ 20123us) server_negotiation.cc:206] Beginning negotiation
0923 20:11:16.074865 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 20:11:16.074894 (+ 29us) server_negotiation.cc:407] Connection header received
0923 20:11:16.074981 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:11:16.074988 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:11:16.075206 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:11:16.075392 (+ 186us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:11:16.077391 (+ 1999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:11:16.078889 (+ 1498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:11:16.081299 (+ 2410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:11:16.083808 (+ 2509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:11:16.083941 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:11:16.084157 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0923 20:11:16.085232 (+ 1075us) server_negotiation.cc:299] Negotiation successful
0923 20:11:16.085391 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19950,"thread_start_us":29441,"threads_started":1}
I20260923 20:11:16.088460 13356 tablet_service.cc:1467] Processing CreateTablet for tablet fcbe9ee8283e4dc3a8f5e461a96956ed (DEFAULT_TABLE table=test-table [id=4976ec40ae874cfa9aa672de95ca0cb4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 20:11:16.090060 13356 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fcbe9ee8283e4dc3a8f5e461a96956ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:11:16.093700 13490 tablet_service.cc:1467] Processing CreateTablet for tablet fcbe9ee8283e4dc3a8f5e461a96956ed (DEFAULT_TABLE table=test-table [id=4976ec40ae874cfa9aa672de95ca0cb4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 20:11:16.095449 13490 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fcbe9ee8283e4dc3a8f5e461a96956ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:11:16.110831 13574 tablet_bootstrap.cc:492] T fcbe9ee8283e4dc3a8f5e461a96956ed P 1a0e5edef160421183713f47489fe44e: Bootstrap starting.
I20260923 20:11:16.112600 13575 tablet_bootstrap.cc:492] T fcbe9ee8283e4dc3a8f5e461a96956ed P 9409360d42ca4922b1c09ee4c93511a4: Bootstrap starting.
I20260923 20:11:16.114176 13576 tablet_bootstrap.cc:492] T fcbe9ee8283e4dc3a8f5e461a96956ed P 54637695ec69409795bd3fba64c7a416: Bootstrap starting.
I20260923 20:11:16.116204 13575 tablet_bootstrap.cc:654] T fcbe9ee8283e4dc3a8f5e461a96956ed P 9409360d42ca4922b1c09ee4c93511a4: Neither blocks nor log segments found. Creating new log.
I20260923 20:11:16.116236 13574 tablet_bootstrap.cc:654] T fcbe9ee8283e4dc3a8f5e461a96956ed P 1a0e5edef160421183713f47489fe44e: Neither blocks nor log segments found. Creating new log.
I20260923 20:11:16.117662 13574 log.cc:824] T fcbe9ee8283e4dc3a8f5e461a96956ed P 1a0e5edef160421183713f47489fe44e: Log is configured to *not* fsync() on all Append() calls
I20260923 20:11:16.118103 13575 log.cc:824] T fcbe9ee8283e4dc3a8f5e461a96956ed P 9409360d42ca4922b1c09ee4c93511a4: Log is configured to *not* fsync() on all Append() calls
I20260923 20:11:16.119205 13576 tablet_bootstrap.cc:654] T fcbe9ee8283e4dc3a8f5e461a96956ed P 54637695ec69409795bd3fba64c7a416: Neither blocks nor log segments found. Creating new log.
I20260923 20:11:16.120453 13576 log.cc:824] T fcbe9ee8283e4dc3a8f5e461a96956ed P 54637695ec69409795bd3fba64c7a416: Log is configured to *not* fsync() on all Append() calls
I20260923 20:11:16.123983 13574 tablet_bootstrap.cc:492] T fcbe9ee8283e4dc3a8f5e461a96956ed P 1a0e5edef160421183713f47489fe44e: No bootstrap required, opened a new log
I20260923 20:11:16.124354 13574 ts_tablet_manager.cc:1397] T fcbe9ee8283e4dc3a8f5e461a96956ed P 1a0e5edef160421183713f47489fe44e: Time spent bootstrapping tablet: real 0.014s user 0.003s sys 0.004s
I20260923 20:11:16.128644 13575 tablet_bootstrap.cc:492] T fcbe9ee8283e4dc3a8f5e461a96956ed P 9409360d42ca4922b1c09ee4c93511a4: No bootstrap required, opened a new log
I20260923 20:11:16.128918 13575 ts_tablet_manager.cc:1397] T fcbe9ee8283e4dc3a8f5e461a96956ed P 9409360d42ca4922b1c09ee4c93511a4: Time spent bootstrapping tablet: real 0.017s user 0.000s sys 0.015s
I20260923 20:11:16.130815 13576 tablet_bootstrap.cc:492] T fcbe9ee8283e4dc3a8f5e461a96956ed P 54637695ec69409795bd3fba64c7a416: No bootstrap required, opened a new log
I20260923 20:11:16.131083 13576 ts_tablet_manager.cc:1397] T fcbe9ee8283e4dc3a8f5e461a96956ed P 54637695ec69409795bd3fba64c7a416: Time spent bootstrapping tablet: real 0.017s user 0.006s sys 0.006s
I20260923 20:11:16.135617 13574 raft_consensus.cc:348] T fcbe9ee8283e4dc3a8f5e461a96956ed P 1a0e5edef160421183713f47489fe44e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a0e5edef160421183713f47489fe44e" member_type: VOTER last_known_addr { host: "127.1.170.129" port: 43565 } } peers { permanent_uuid: "9409360d42ca4922b1c09ee4c93511a4" member_type: VOTER last_known_addr { host: "127.1.170.130" port: 36713 } } peers { permanent_uuid: "54637695ec69409795bd3fba64c7a416" member_type: VOTER last_known_addr { host: "127.1.170.131" port: 33467 } }
I20260923 20:11:16.136090 13574 raft_consensus.cc:374] T fcbe9ee8283e4dc3a8f5e461a96956ed P 1a0e5edef160421183713f47489fe44e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:11:16.136180 13574 raft_consensus.cc:729] T fcbe9ee8283e4dc3a8f5e461a96956ed P 1a0e5edef160421183713f47489fe44e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1a0e5edef160421183713f47489fe44e, State: Initialized, Role: FOLLOWER
I20260923 20:11:16.136813 13574 consensus_queue.cc:260] T fcbe9ee8283e4dc3a8f5e461a96956ed P 1a0e5edef160421183713f47489fe44e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a0e5edef160421183713f47489fe44e" member_type: VOTER last_known_addr { host: "127.1.170.129" port: 43565 } } peers { permanent_uuid: "9409360d42ca4922b1c09ee4c93511a4" member_type: VOTER last_known_addr { host: "127.1.170.130" port: 36713 } } peers { permanent_uuid: "54637695ec69409795bd3fba64c7a416" member_type: VOTER last_known_addr { host: "127.1.170.131" port: 33467 } }
I20260923 20:11:16.138289 13575 raft_consensus.cc:348] T fcbe9ee8283e4dc3a8f5e461a96956ed P 9409360d42ca4922b1c09ee4c93511a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a0e5edef160421183713f47489fe44e" member_type: VOTER last_known_addr { host: "127.1.170.129" port: 43565 } } peers { permanent_uuid: "9409360d42ca4922b1c09ee4c93511a4" member_type: VOTER last_known_addr { host: "127.1.170.130" port: 36713 } } peers { permanent_uuid: "54637695ec69409795bd3fba64c7a416" member_type: VOTER last_known_addr { host: "127.1.170.131" port: 33467 } }
I20260923 20:11:16.138684 13575 raft_consensus.cc:374] T fcbe9ee8283e4dc3a8f5e461a96956ed P 9409360d42ca4922b1c09ee4c93511a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:11:16.138780 13575 raft_consensus.cc:729] T fcbe9ee8283e4dc3a8f5e461a96956ed P 9409360d42ca4922b1c09ee4c93511a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9409360d42ca4922b1c09ee4c93511a4, State: Initialized, Role: FOLLOWER
I20260923 20:11:16.138990 13574 ts_tablet_manager.cc:1428] T fcbe9ee8283e4dc3a8f5e461a96956ed P 1a0e5edef160421183713f47489fe44e: Time spent starting tablet: real 0.014s user 0.015s sys 0.001s
I20260923 20:11:16.139618 13575 consensus_queue.cc:260] T fcbe9ee8283e4dc3a8f5e461a96956ed P 9409360d42ca4922b1c09ee4c93511a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a0e5edef160421183713f47489fe44e" member_type: VOTER last_known_addr { host: "127.1.170.129" port: 43565 } } peers { permanent_uuid: "9409360d42ca4922b1c09ee4c93511a4" member_type: VOTER last_known_addr { host: "127.1.170.130" port: 36713 } } peers { permanent_uuid: "54637695ec69409795bd3fba64c7a416" member_type: VOTER last_known_addr { host: "127.1.170.131" port: 33467 } }
I20260923 20:11:16.140753 13421 heartbeater.cc:503] Master 127.1.170.190:46541 was elected leader, sending a full tablet report...
I20260923 20:11:16.141243 13575 ts_tablet_manager.cc:1428] T fcbe9ee8283e4dc3a8f5e461a96956ed P 9409360d42ca4922b1c09ee4c93511a4: Time spent starting tablet: real 0.012s user 0.004s sys 0.007s
I20260923 20:11:16.141717 13576 raft_consensus.cc:348] T fcbe9ee8283e4dc3a8f5e461a96956ed P 54637695ec69409795bd3fba64c7a416 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a0e5edef160421183713f47489fe44e" member_type: VOTER last_known_addr { host: "127.1.170.129" port: 43565 } } peers { permanent_uuid: "9409360d42ca4922b1c09ee4c93511a4" member_type: VOTER last_known_addr { host: "127.1.170.130" port: 36713 } } peers { permanent_uuid: "54637695ec69409795bd3fba64c7a416" member_type: VOTER last_known_addr { host: "127.1.170.131" port: 33467 } }
I20260923 20:11:16.142199 13576 raft_consensus.cc:374] T fcbe9ee8283e4dc3a8f5e461a96956ed P 54637695ec69409795bd3fba64c7a416 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:11:16.142298 13576 raft_consensus.cc:729] T fcbe9ee8283e4dc3a8f5e461a96956ed P 54637695ec69409795bd3fba64c7a416 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 54637695ec69409795bd3fba64c7a416, State: Initialized, Role: FOLLOWER
I20260923 20:11:16.142894 13576 consensus_queue.cc:260] T fcbe9ee8283e4dc3a8f5e461a96956ed P 54637695ec69409795bd3fba64c7a416 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a0e5edef160421183713f47489fe44e" member_type: VOTER last_known_addr { host: "127.1.170.129" port: 43565 } } peers { permanent_uuid: "9409360d42ca4922b1c09ee4c93511a4" member_type: VOTER last_known_addr { host: "127.1.170.130" port: 36713 } } peers { permanent_uuid: "54637695ec69409795bd3fba64c7a416" member_type: VOTER last_known_addr { host: "127.1.170.131" port: 33467 } }
I20260923 20:11:16.144229 13555 heartbeater.cc:503] Master 127.1.170.190:46541 was elected leader, sending a full tablet report...
I20260923 20:11:16.144556 13576 ts_tablet_manager.cc:1428] T fcbe9ee8283e4dc3a8f5e461a96956ed P 54637695ec69409795bd3fba64c7a416: Time spent starting tablet: real 0.013s user 0.010s sys 0.004s
W20260923 20:11:16.212704 13556 tablet.cc:2369] T fcbe9ee8283e4dc3a8f5e461a96956ed P 54637695ec69409795bd3fba64c7a416: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 20:11:16.302143 13581 raft_consensus.cc:482] T fcbe9ee8283e4dc3a8f5e461a96956ed P 9409360d42ca4922b1c09ee4c93511a4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 20:11:16.302481 13581 raft_consensus.cc:504] T fcbe9ee8283e4dc3a8f5e461a96956ed P 9409360d42ca4922b1c09ee4c93511a4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a0e5edef160421183713f47489fe44e" member_type: VOTER last_known_addr { host: "127.1.170.129" port: 43565 } } peers { permanent_uuid: "9409360d42ca4922b1c09ee4c93511a4" member_type: VOTER last_known_addr { host: "127.1.170.130" port: 36713 } } peers { permanent_uuid: "54637695ec69409795bd3fba64c7a416" member_type: VOTER last_known_addr { host: "127.1.170.131" port: 33467 } }
I20260923 20:11:16.304306 13581 leader_election.cc:288] T fcbe9ee8283e4dc3a8f5e461a96956ed P 9409360d42ca4922b1c09ee4c93511a4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1a0e5edef160421183713f47489fe44e (127.1.170.129:43565), 54637695ec69409795bd3fba64c7a416 (127.1.170.131:33467)
W20260923 20:11:16.310546 13422 tablet.cc:2369] T fcbe9ee8283e4dc3a8f5e461a96956ed P 9409360d42ca4922b1c09ee4c93511a4: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260923 20:11:16.311635 13279 tablet.cc:2369] T fcbe9ee8283e4dc3a8f5e461a96956ed P 1a0e5edef160421183713f47489fe44e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 20:11:16.314998 13570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:11:16.304640 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.170.130:60719 (local address 127.1.170.129:43565)
0923 20:11:16.304888 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0923 20:11:16.304894 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:11:16.305831 (+ 937us) server_negotiation.cc:407] Connection header received
0923 20:11:16.306080 (+ 249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:11:16.306086 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:11:16.306168 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:11:16.306261 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:11:16.307516 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:11:16.308461 (+ 945us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:11:16.312671 (+ 4210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:11:16.313781 (+ 1110us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:11:16.313944 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:11:16.314019 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0923 20:11:16.314610 (+ 591us) server_negotiation.cc:299] Negotiation successful
0923 20:11:16.314752 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260923 20:11:16.315424 13571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:11:16.304807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.170.130:41875 (local address 127.1.170.131:33467)
0923 20:11:16.305027 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0923 20:11:16.305034 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:11:16.305974 (+ 940us) server_negotiation.cc:407] Connection header received
0923 20:11:16.306178 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:11:16.306183 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:11:16.306259 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:11:16.306337 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:11:16.307636 (+ 1299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:11:16.308678 (+ 1042us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:11:16.312711 (+ 4033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:11:16.314268 (+ 1557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:11:16.314449 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:11:16.314539 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0923 20:11:16.314851 (+ 312us) server_negotiation.cc:299] Negotiation successful
0923 20:11:16.314983 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260923 20:11:16.315428 13584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:11:16.304593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.131:33467 (local address 127.1.170.130:41875)
0923 20:11:16.305821 (+ 1228us) negotiation.cc:107] Waiting for socket to connect
0923 20:11:16.305843 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0923 20:11:16.306030 (+ 187us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:11:16.306489 (+ 459us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:11:16.306492 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:11:16.306515 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:11:16.307472 (+ 957us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:11:16.307486 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:11:16.308817 (+ 1331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:11:16.308823 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:11:16.312479 (+ 3656us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:11:16.312492 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:11:16.314500 (+ 2008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:11:16.314509 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:11:16.314680 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:11:16.314725 (+ 45us) client_negotiation.cc:769] Sending connection context
0923 20:11:16.314816 (+ 91us) client_negotiation.cc:240] Negotiation successful
0923 20:11:16.314942 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1044,"mutex_wait_us":940,"thread_start_us":817,"threads_started":1}
I20260923 20:11:16.315837 13583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:11:16.304536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.129:43565 (local address 127.1.170.130:60719)
0923 20:11:16.305703 (+ 1167us) negotiation.cc:107] Waiting for socket to connect
0923 20:11:16.305729 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0923 20:11:16.305948 (+ 219us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:11:16.306298 (+ 350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:11:16.306305 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:11:16.306333 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:11:16.307344 (+ 1011us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:11:16.307362 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:11:16.308607 (+ 1245us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:11:16.308617 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:11:16.312476 (+ 3859us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:11:16.312492 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:11:16.314219 (+ 1727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:11:16.314224 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:11:16.314399 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:11:16.314446 (+ 47us) client_negotiation.cc:769] Sending connection context
0923 20:11:16.315215 (+ 769us) client_negotiation.cc:240] Negotiation successful
0923 20:11:16.315354 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":986,"mutex_wait_us":920,"thread_start_us":667,"threads_started":1}
I20260923 20:11:16.316538 13510 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcbe9ee8283e4dc3a8f5e461a96956ed" candidate_uuid: "9409360d42ca4922b1c09ee4c93511a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54637695ec69409795bd3fba64c7a416" is_pre_election: true
I20260923 20:11:16.317023 13510 raft_consensus.cc:2454] T fcbe9ee8283e4dc3a8f5e461a96956ed P 54637695ec69409795bd3fba64c7a416 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9409360d42ca4922b1c09ee4c93511a4 in term 0.
I20260923 20:11:16.317670 13309 leader_election.cc:302] T fcbe9ee8283e4dc3a8f5e461a96956ed P 9409360d42ca4922b1c09ee4c93511a4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 54637695ec69409795bd3fba64c7a416, 9409360d42ca4922b1c09ee4c93511a4; no voters:
I20260923 20:11:16.318001 13233 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcbe9ee8283e4dc3a8f5e461a96956ed" candidate_uuid: "9409360d42ca4922b1c09ee4c93511a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1a0e5edef160421183713f47489fe44e" is_pre_election: true
I20260923 20:11:16.318503 13233 raft_consensus.cc:2454] T fcbe9ee8283e4dc3a8f5e461a96956ed P 1a0e5edef160421183713f47489fe44e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9409360d42ca4922b1c09ee4c93511a4 in term 0.
I20260923 20:11:16.318580 13581 raft_consensus.cc:2781] T fcbe9ee8283e4dc3a8f5e461a96956ed P 9409360d42ca4922b1c09ee4c93511a4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 20:11:16.318722 13581 raft_consensus.cc:482] T fcbe9ee8283e4dc3a8f5e461a96956ed P 9409360d42ca4922b1c09ee4c93511a4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 20:11:16.318846 13581 raft_consensus.cc:3037] T fcbe9ee8283e4dc3a8f5e461a96956ed P 9409360d42ca4922b1c09ee4c93511a4 [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:11:16.321133 13581 raft_consensus.cc:504] T fcbe9ee8283e4dc3a8f5e461a96956ed P 9409360d42ca4922b1c09ee4c93511a4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a0e5edef160421183713f47489fe44e" member_type: VOTER last_known_addr { host: "127.1.170.129" port: 43565 } } peers { permanent_uuid: "9409360d42ca4922b1c09ee4c93511a4" member_type: VOTER last_known_addr { host: "127.1.170.130" port: 36713 } } peers { permanent_uuid: "54637695ec69409795bd3fba64c7a416" member_type: VOTER last_known_addr { host: "127.1.170.131" port: 33467 } }
I20260923 20:11:16.321977 13581 leader_election.cc:288] T fcbe9ee8283e4dc3a8f5e461a96956ed P 9409360d42ca4922b1c09ee4c93511a4 [CANDIDATE]: Term 1 election: Requested vote from peers 1a0e5edef160421183713f47489fe44e (127.1.170.129:43565), 54637695ec69409795bd3fba64c7a416 (127.1.170.131:33467)
I20260923 20:11:16.322218 13233 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcbe9ee8283e4dc3a8f5e461a96956ed" candidate_uuid: "9409360d42ca4922b1c09ee4c93511a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1a0e5edef160421183713f47489fe44e"
I20260923 20:11:16.322463 13233 raft_consensus.cc:3037] T fcbe9ee8283e4dc3a8f5e461a96956ed P 1a0e5edef160421183713f47489fe44e [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:11:16.322458 13510 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcbe9ee8283e4dc3a8f5e461a96956ed" candidate_uuid: "9409360d42ca4922b1c09ee4c93511a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54637695ec69409795bd3fba64c7a416"
I20260923 20:11:16.322652 13510 raft_consensus.cc:3037] T fcbe9ee8283e4dc3a8f5e461a96956ed P 54637695ec69409795bd3fba64c7a416 [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:11:16.325815 13510 raft_consensus.cc:2454] T fcbe9ee8283e4dc3a8f5e461a96956ed P 54637695ec69409795bd3fba64c7a416 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9409360d42ca4922b1c09ee4c93511a4 in term 1.
I20260923 20:11:16.325815 13233 raft_consensus.cc:2454] T fcbe9ee8283e4dc3a8f5e461a96956ed P 1a0e5edef160421183713f47489fe44e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9409360d42ca4922b1c09ee4c93511a4 in term 1.
I20260923 20:11:16.326324 13311 leader_election.cc:302] T fcbe9ee8283e4dc3a8f5e461a96956ed P 9409360d42ca4922b1c09ee4c93511a4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1a0e5edef160421183713f47489fe44e, 9409360d42ca4922b1c09ee4c93511a4; no voters:
I20260923 20:11:16.326654 13581 raft_consensus.cc:2781] T fcbe9ee8283e4dc3a8f5e461a96956ed P 9409360d42ca4922b1c09ee4c93511a4 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 20:11:16.327162 13581 raft_consensus.cc:686] T fcbe9ee8283e4dc3a8f5e461a96956ed P 9409360d42ca4922b1c09ee4c93511a4 [term 1 LEADER]: Becoming Leader. State: Replica: 9409360d42ca4922b1c09ee4c93511a4, State: Running, Role: LEADER
I20260923 20:11:16.327623 13581 consensus_queue.cc:237] T fcbe9ee8283e4dc3a8f5e461a96956ed P 9409360d42ca4922b1c09ee4c93511a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a0e5edef160421183713f47489fe44e" member_type: VOTER last_known_addr { host: "127.1.170.129" port: 43565 } } peers { permanent_uuid: "9409360d42ca4922b1c09ee4c93511a4" member_type: VOTER last_known_addr { host: "127.1.170.130" port: 36713 } } peers { permanent_uuid: "54637695ec69409795bd3fba64c7a416" member_type: VOTER last_known_addr { host: "127.1.170.131" port: 33467 } }
I20260923 20:11:16.332329 13063 catalog_manager.cc:5495] T fcbe9ee8283e4dc3a8f5e461a96956ed P 9409360d42ca4922b1c09ee4c93511a4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9409360d42ca4922b1c09ee4c93511a4 (127.1.170.130). New cstate: current_term: 1 leader_uuid: "9409360d42ca4922b1c09ee4c93511a4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a0e5edef160421183713f47489fe44e" member_type: VOTER last_known_addr { host: "127.1.170.129" port: 43565 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9409360d42ca4922b1c09ee4c93511a4" member_type: VOTER last_known_addr { host: "127.1.170.130" port: 36713 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "54637695ec69409795bd3fba64c7a416" member_type: VOTER last_known_addr { host: "127.1.170.131" port: 33467 } health_report { overall_health: UNKNOWN } } }
I20260923 20:11:16.358268 13573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:11:16.352941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48034 (local address 127.1.170.130:36713)
0923 20:11:16.353141 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0923 20:11:16.353148 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:11:16.353179 (+ 31us) server_negotiation.cc:407] Connection header received
0923 20:11:16.353285 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:11:16.353291 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:11:16.353360 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:11:16.353511 (+ 151us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 20:11:16.353870 (+ 359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:11:16.354686 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:11:16.356142 (+ 1456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:11:16.356562 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:11:16.356691 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:11:16.357139 (+ 448us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 20:11:16.357618 (+ 479us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 20:11:16.357773 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0923 20:11:16.357932 (+ 159us) server_negotiation.cc:299] Negotiation successful
0923 20:11:16.358008 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260923 20:11:16.366403 13510 raft_consensus.cc:1261] T fcbe9ee8283e4dc3a8f5e461a96956ed P 54637695ec69409795bd3fba64c7a416 [term 1 FOLLOWER]: Refusing update from remote peer 9409360d42ca4922b1c09ee4c93511a4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 20:11:16.366405 13233 raft_consensus.cc:1261] T fcbe9ee8283e4dc3a8f5e461a96956ed P 1a0e5edef160421183713f47489fe44e [term 1 FOLLOWER]: Refusing update from remote peer 9409360d42ca4922b1c09ee4c93511a4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 20:11:16.367412 13581 consensus_queue.cc:1035] T fcbe9ee8283e4dc3a8f5e461a96956ed P 9409360d42ca4922b1c09ee4c93511a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1a0e5edef160421183713f47489fe44e" member_type: VOTER last_known_addr { host: "127.1.170.129" port: 43565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 20:11:16.367707 13585 consensus_queue.cc:1035] T fcbe9ee8283e4dc3a8f5e461a96956ed P 9409360d42ca4922b1c09ee4c93511a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "54637695ec69409795bd3fba64c7a416" member_type: VOTER last_known_addr { host: "127.1.170.131" port: 33467 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 20:11:16.378926 13588 mvcc.cc:205] Tried to move back new op lower bound from 7332635755986464768 to 7332635755841720320. Current Snapshot: MvccSnapshot[applied={T|T < 7332635755986464768}]
I20260923 20:11:16.395784 13064 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47708:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 20:11:16.396093 13064 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47708:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 20:11:16.400169 13064 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c18f4886aae44650bc0fad2c61749e75: Sending DeleteTablet for 3 replicas of tablet fcbe9ee8283e4dc3a8f5e461a96956ed
I20260923 20:11:16.401446 13490 tablet_service.cc:1514] Processing DeleteTablet for tablet fcbe9ee8283e4dc3a8f5e461a96956ed with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 20:11:16 UTC) from {username='kudu', principal='kudu/127.1.170.190@KRBTEST.COM'} at 127.0.0.1:56224
I20260923 20:11:16.401566 13356 tablet_service.cc:1514] Processing DeleteTablet for tablet fcbe9ee8283e4dc3a8f5e461a96956ed with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 20:11:16 UTC) from {username='kudu', principal='kudu/127.1.170.190@KRBTEST.COM'} at 127.0.0.1:48020
I20260923 20:11:16.401795 13213 tablet_service.cc:1514] Processing DeleteTablet for tablet fcbe9ee8283e4dc3a8f5e461a96956ed with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 20:11:16 UTC) from {username='kudu', principal='kudu/127.1.170.190@KRBTEST.COM'} at 127.0.0.1:37916
I20260923 20:11:16.408041 1706 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskqUuku6/build/asan/bin/kudu pid 13145
I20260923 20:11:16.414284 13602 tablet_replica.cc:330] T fcbe9ee8283e4dc3a8f5e461a96956ed P 1a0e5edef160421183713f47489fe44e: stopping tablet replica
I20260923 20:11:16.414767 13602 raft_consensus.cc:2229] T fcbe9ee8283e4dc3a8f5e461a96956ed P 1a0e5edef160421183713f47489fe44e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 20:11:16.415297 13602 raft_consensus.cc:2258] T fcbe9ee8283e4dc3a8f5e461a96956ed P 1a0e5edef160421183713f47489fe44e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 20:11:16.415295 13600 tablet_replica.cc:330] T fcbe9ee8283e4dc3a8f5e461a96956ed P 9409360d42ca4922b1c09ee4c93511a4: stopping tablet replica
I20260923 20:11:16.415479 13601 tablet_replica.cc:330] T fcbe9ee8283e4dc3a8f5e461a96956ed P 54637695ec69409795bd3fba64c7a416: stopping tablet replica
I20260923 20:11:16.415997 13601 raft_consensus.cc:2229] T fcbe9ee8283e4dc3a8f5e461a96956ed P 54637695ec69409795bd3fba64c7a416 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 20:11:16.416574 13600 raft_consensus.cc:2229] T fcbe9ee8283e4dc3a8f5e461a96956ed P 9409360d42ca4922b1c09ee4c93511a4 [term 1 LEADER]: Raft consensus shutting down.
I20260923 20:11:16.417173 13601 raft_consensus.cc:2258] T fcbe9ee8283e4dc3a8f5e461a96956ed P 54637695ec69409795bd3fba64c7a416 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 20:11:16.417546 13602 ts_tablet_manager.cc:1905] T fcbe9ee8283e4dc3a8f5e461a96956ed P 1a0e5edef160421183713f47489fe44e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 20:11:16.418073 13600 raft_consensus.cc:2258] T fcbe9ee8283e4dc3a8f5e461a96956ed P 9409360d42ca4922b1c09ee4c93511a4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 20:11:16.421801 13600 ts_tablet_manager.cc:1905] T fcbe9ee8283e4dc3a8f5e461a96956ed P 9409360d42ca4922b1c09ee4c93511a4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 20:11:16.425037 13602 ts_tablet_manager.cc:1918] T fcbe9ee8283e4dc3a8f5e461a96956ed P 1a0e5edef160421183713f47489fe44e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 20:11:16.425257 13602 log.cc:1196] T fcbe9ee8283e4dc3a8f5e461a96956ed P 1a0e5edef160421183713f47489fe44e: Deleting WAL directory at /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790194121577027-1706-0/minicluster-data/ts-0/wal/wals/fcbe9ee8283e4dc3a8f5e461a96956ed
I20260923 20:11:16.425843 13602 ts_tablet_manager.cc:1939] T fcbe9ee8283e4dc3a8f5e461a96956ed P 1a0e5edef160421183713f47489fe44e: Deleting consensus metadata
I20260923 20:11:16.426673 13601 ts_tablet_manager.cc:1905] T fcbe9ee8283e4dc3a8f5e461a96956ed P 54637695ec69409795bd3fba64c7a416: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 20:11:16.427727 13050 catalog_manager.cc:4841] TS 1a0e5edef160421183713f47489fe44e (127.1.170.129:43565): tablet fcbe9ee8283e4dc3a8f5e461a96956ed (table test-table [id=4976ec40ae874cfa9aa672de95ca0cb4]) successfully deleted
I20260923 20:11:16.430763 13600 ts_tablet_manager.cc:1918] T fcbe9ee8283e4dc3a8f5e461a96956ed P 9409360d42ca4922b1c09ee4c93511a4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 20:11:16.430984 13600 log.cc:1196] T fcbe9ee8283e4dc3a8f5e461a96956ed P 9409360d42ca4922b1c09ee4c93511a4: Deleting WAL directory at /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790194121577027-1706-0/minicluster-data/ts-1/wal/wals/fcbe9ee8283e4dc3a8f5e461a96956ed
I20260923 20:11:16.431591 13600 ts_tablet_manager.cc:1939] T fcbe9ee8283e4dc3a8f5e461a96956ed P 9409360d42ca4922b1c09ee4c93511a4: Deleting consensus metadata
Sep 23 20:11:16 dist-test-slave-th4g krb5kdc[12588](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194272, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.170.129@KRBTEST.COM
I20260923 20:11:16.433473 13050 catalog_manager.cc:4841] TS 9409360d42ca4922b1c09ee4c93511a4 (127.1.170.130:36713): tablet fcbe9ee8283e4dc3a8f5e461a96956ed (table test-table [id=4976ec40ae874cfa9aa672de95ca0cb4]) successfully deleted
I20260923 20:11:16.433688 13601 ts_tablet_manager.cc:1918] T fcbe9ee8283e4dc3a8f5e461a96956ed P 54637695ec69409795bd3fba64c7a416: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 20:11:16.433853 13601 log.cc:1196] T fcbe9ee8283e4dc3a8f5e461a96956ed P 54637695ec69409795bd3fba64c7a416: Deleting WAL directory at /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790194121577027-1706-0/minicluster-data/ts-2/wal/wals/fcbe9ee8283e4dc3a8f5e461a96956ed
I20260923 20:11:16.434355 13601 ts_tablet_manager.cc:1939] T fcbe9ee8283e4dc3a8f5e461a96956ed P 54637695ec69409795bd3fba64c7a416: Deleting consensus metadata
I20260923 20:11:16.435988 13048 catalog_manager.cc:4841] TS 54637695ec69409795bd3fba64c7a416 (127.1.170.131:33467): tablet fcbe9ee8283e4dc3a8f5e461a96956ed (table test-table [id=4976ec40ae874cfa9aa672de95ca0cb4]) successfully deleted
I20260923 20:11:16.440248 13570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:11:16.409432 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37926 (local address 127.1.170.129:43565)
0923 20:11:16.409676 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0923 20:11:16.409686 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0923 20:11:16.421382 (+ 11696us) server_negotiation.cc:407] Connection header received
0923 20:11:16.421907 (+ 525us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:11:16.421917 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:11:16.422020 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:11:16.422097 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:11:16.424031 (+ 1934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:11:16.425142 (+ 1111us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:11:16.426986 (+ 1844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:11:16.427654 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:11:16.427763 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:11:16.433835 (+ 6072us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:11:16.433854 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:11:16.433889 (+ 35us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:11:16.433933 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:11:16.437393 (+ 3460us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:11:16.438267 (+ 874us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:11:16.438274 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:11:16.438276 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:11:16.438392 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:11:16.438760 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:11:16.438763 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:11:16.438765 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:11:16.439080 (+ 315us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:11:16.439357 (+ 277us) server_negotiation.cc:1035] Waiting for connection context
0923 20:11:16.439619 (+ 262us) server_negotiation.cc:299] Negotiation successful
0923 20:11:16.439865 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260923 20:11:16.440994 13273 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:11:16.531666 1706 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqUuku6/build/asan/bin/kudu with pid 13145
I20260923 20:11:16.549705 1706 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskqUuku6/build/asan/bin/kudu pid 13284
Sep 23 20:11:16 dist-test-slave-th4g krb5kdc[12588](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194272, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.170.130@KRBTEST.COM
I20260923 20:11:16.566895 13573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:11:16.550534 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48048 (local address 127.1.170.130:36713)
0923 20:11:16.550725 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0923 20:11:16.550731 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:11:16.550753 (+ 22us) server_negotiation.cc:407] Connection header received
0923 20:11:16.550945 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:11:16.550951 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:11:16.551049 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:11:16.551178 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:11:16.552442 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:11:16.553706 (+ 1264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:11:16.555051 (+ 1345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:11:16.555739 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:11:16.555865 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:11:16.559916 (+ 4051us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:11:16.559932 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:11:16.559956 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:11:16.560001 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:11:16.563899 (+ 3898us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:11:16.564660 (+ 761us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:11:16.564669 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:11:16.564673 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:11:16.564874 (+ 201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:11:16.565286 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:11:16.565292 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:11:16.565296 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:11:16.565728 (+ 432us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:11:16.566082 (+ 354us) server_negotiation.cc:1035] Waiting for connection context
0923 20:11:16.566255 (+ 173us) server_negotiation.cc:299] Negotiation successful
0923 20:11:16.566477 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260923 20:11:16.567615 13416 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:11:16.679764 1706 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqUuku6/build/asan/bin/kudu with pid 13284
I20260923 20:11:16.698524 1706 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskqUuku6/build/asan/bin/kudu pid 13427
Sep 23 20:11:16 dist-test-slave-th4g krb5kdc[12588](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194272, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.170.131@KRBTEST.COM
I20260923 20:11:16.714955 13571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:11:16.699310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56236 (local address 127.1.170.131:33467)
0923 20:11:16.699481 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0923 20:11:16.699486 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 20:11:16.699503 (+ 17us) server_negotiation.cc:407] Connection header received
0923 20:11:16.699660 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:11:16.699663 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:11:16.699738 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:11:16.699820 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:11:16.701352 (+ 1532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:11:16.702296 (+ 944us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:11:16.704179 (+ 1883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:11:16.704552 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:11:16.704642 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:11:16.709499 (+ 4857us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:11:16.709511 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:11:16.709528 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:11:16.709563 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:11:16.712191 (+ 2628us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:11:16.712768 (+ 577us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:11:16.712775 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:11:16.712777 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:11:16.712897 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:11:16.713254 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:11:16.713257 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:11:16.713259 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:11:16.713591 (+ 332us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:11:16.713883 (+ 292us) server_negotiation.cc:1035] Waiting for connection context
0923 20:11:16.714297 (+ 414us) server_negotiation.cc:299] Negotiation successful
0923 20:11:16.714543 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260923 20:11:16.715592 13550 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:11:16.806107 1706 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqUuku6/build/asan/bin/kudu with pid 13427
I20260923 20:11:16.822490 1706 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskqUuku6/build/asan/bin/kudu pid 13032
I20260923 20:11:16.823625 13094 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:11:16.934450 1706 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqUuku6/build/asan/bin/kudu with pid 13032
I20260923 20:11:16.955845 1706 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260923 20:11:17.294939 1706 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260923 20:11:17.295261 1706 mini_ranger.cc:66] Stopping Ranger...
I20260923 20:11:17.640906 1706 mini_ranger.cc:68] Stopped Ranger
2026-09-23 20:11:17.641 UTC [12618] LOG: received smart shutdown request
2026-09-23 20:11:17.645 UTC [12618] LOG: background worker "logical replication launcher" (PID 12627) exited with exit code 1
2026-09-23 20:11:17.646 UTC [12622] LOG: shutting down
2026-09-23 20:11:17.829 UTC [12618] LOG: database system is shut down
2026-09-23T20:11:17Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (52481 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194121577027-1706-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 20:11:18 dist-test-slave-th4g krb5kdc[13616](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 20:11:18 dist-test-slave-th4g krb5kdc[13616](info): set up 2 sockets
Sep 23 20:11:18 dist-test-slave-th4g krb5kdc[13616](info): commencing operation
krb5kdc: starting...
W20260923 20:11:19.078887 1706 mini_kdc.cc:121] Time spent starting KDC: real 1.080s user 0.003s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194121577027-1706-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194121577027-1706-0/krb5kdc/test-user.keytab.
Sep 23 20:11:19 dist-test-slave-th4g krb5kdc[13616](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194279, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T20:11:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T20:11:19Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.1.170.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.1.170.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.1.170.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194121577027-1706-0/krb5kdc/rangeradmin_127.1.170.148@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.1.170.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194121577027-1706-0/krb5kdc/rangeradmin_127.1.170.148@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.1.170.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.1.170.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.1.170.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194121577027-1706-0/krb5kdc/rangerlookup_127.1.170.148@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.1.170.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194121577027-1706-0/krb5kdc/rangerlookup_127.1.170.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.1.170.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.1.170.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.1.170.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194121577027-1706-0/krb5kdc/HTTP_127.1.170.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.1.170.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194121577027-1706-0/krb5kdc/HTTP_127.1.170.148@KRBTEST.COM.keytab.
I20260923 20:11:19.423208 1706 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194121577027-1706-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskqUuku6/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194121577027-1706-0/minicluster-data/postgres -l logfile start
2026-09-23 20:11:22.117 UTC [13646] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-23 20:11:22.117 UTC [13646] LOG: listening on IPv4 address "127.1.170.148", port 59805
2026-09-23 20:11:22.120 UTC [13646] LOG: listening on Unix socket "/tmp/.s.PGSQL.59805"
2026-09-23 20:11:22.142 UTC [13649] LOG: database system was shut down at 2026-09-23 20:11:20 UTC
2026-09-23 20:11:22.147 UTC [13646] LOG: database system is ready to accept connections
I20260923 20:11:23.118185 1706 mini_postgres.cc:96] Postgres bound to 59805
2026-09-23 20:11:23.135 UTC [13657] FATAL: database "slave" does not exist
127.1.170.148:59805 - accepting connections
I20260923 20:11:23.135733 1706 mini_ranger.cc:161] Starting Ranger...
I20260923 20:11:23.162782 1706 mini_ranger.cc:84] Created miniranger Postgres user
I20260923 20:11:23.748131 1706 mini_ranger.cc:87] Created ranger Postgres database
I20260923 20:11:23.748265 1706 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194121577027-1706-0/minicluster-data/ranger-admin
2026-09-23 20:11:24,052 [I] DB FLAVOR :POSTGRES
2026-09-23 20:11:24,053 [I] --------- Verifying Ranger DB connection ---------
2026-09-23 20:11:24,053 [I] Checking connection..
2026-09-23 20:11:24,053 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskqUuku6/build/asan/bin/postgresql.jar:/tmp/dist-test-taskqUuku6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.170.148:59805/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-23 20:11:24,297 [I] Checking connection passed.
2026-09-23 20:11:24,297 [I] --------- Verifying version history table ---------
2026-09-23 20:11:24,297 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskqUuku6/build/asan/bin/postgresql.jar:/tmp/dist-test-taskqUuku6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.170.148:59805/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-23 20:11:24,549 [I] Table x_db_version_h does not exist in database ranger
2026-09-23 20:11:24,550 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskqUuku6/build/asan/bin/postgresql.jar:/tmp/dist-test-taskqUuku6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.170.148:59805/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-23 20:11:24,806 [I] Table x_db_version_h does not exist in database ranger
2026-09-23 20:11:24,807 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-23 20:11:24,807 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskqUuku6/build/asan/bin/postgresql.jar:/tmp/dist-test-taskqUuku6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.170.148:59805/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskqUuku6/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-23 20:11:25.055 UTC [13721] WARNING: there is no transaction in progress
2026-09-23 20:11:25,061 [I] create_dbversion_catalog.sql file imported successfully
2026-09-23 20:11:25,061 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskqUuku6/build/asan/bin/postgresql.jar:/tmp/dist-test-taskqUuku6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.170.148:59805/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-23 20:11:25,326 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-23 20:11:25,326 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-23 20:11:25,326 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskqUuku6/build/asan/bin/postgresql.jar:/tmp/dist-test-taskqUuku6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.170.148:59805/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-23 20:11:25,572 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskqUuku6/build/asan/bin/postgresql.jar:/tmp/dist-test-taskqUuku6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.170.148:59805/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-23 20:11:25,816 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskqUuku6/build/asan/bin/postgresql.jar:/tmp/dist-test-taskqUuku6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.170.148:59805/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-th4g.c.gcp-upstream.internal','N') ;"
2026-09-23 20:11:26,064 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskqUuku6/build/asan/bin/postgresql.jar:/tmp/dist-test-taskqUuku6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.170.148:59805/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-23 20:11:26,333 [I] Table x_portal_user does not exist in database ranger
2026-09-23 20:11:26,334 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskqUuku6/build/asan/bin/postgresql.jar:/tmp/dist-test-taskqUuku6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.170.148:59805/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-23 20:11:26,582 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-23 20:11:26,582 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskqUuku6/build/asan/bin/postgresql.jar:/tmp/dist-test-taskqUuku6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.170.148:59805/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-23 20:11:26,827 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskqUuku6/build/asan/bin/postgresql.jar:/tmp/dist-test-taskqUuku6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.170.148:59805/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-23 20:11:27,065 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-23 20:11:27,065 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskqUuku6/build/asan/bin/postgresql.jar:/tmp/dist-test-taskqUuku6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.170.148:59805/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskqUuku6/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-23 20:11:28.158 UTC [13829] WARNING: there is no transaction in progress
2026-09-23 20:11:28.176 UTC [13829] WARNING: there is no transaction in progress
2026-09-23 20:11:28.193 UTC [13829] WARNING: there is no transaction in progress
2026-09-23 20:11:28.209 UTC [13829] WARNING: there is no transaction in progress
2026-09-23 20:11:28.229 UTC [13829] WARNING: there is no transaction in progress
2026-09-23 20:11:28.248 UTC [13829] WARNING: there is no transaction in progress
2026-09-23 20:11:28.312 UTC [13829] WARNING: there is no transaction in progress
2026-09-23 20:11:28.319 UTC [13829] WARNING: there is no transaction in progress
2026-09-23 20:11:28.334 UTC [13829] WARNING: there is no transaction in progress
2026-09-23 20:11:28.346 UTC [13829] WARNING: there is no transaction in progress
2026-09-23 20:11:28.356 UTC [13829] WARNING: there is no transaction in progress
2026-09-23 20:11:28.370 UTC [13829] WARNING: there is no transaction in progress
2026-09-23 20:11:28.380 UTC [13829] WARNING: there is no transaction in progress
2026-09-23 20:11:28.386 UTC [13829] WARNING: there is no transaction in progress
2026-09-23 20:11:28.394 UTC [13829] WARNING: there is no transaction in progress
2026-09-23 20:11:28.401 UTC [13829] WARNING: there is no transaction in progress
2026-09-23 20:11:29,089 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-23 20:11:29,089 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskqUuku6/build/asan/bin/postgresql.jar:/tmp/dist-test-taskqUuku6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.170.148:59805/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-23 20:11:29,322 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-23 20:11:29,322 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskqUuku6/build/asan/bin/postgresql.jar:/tmp/dist-test-taskqUuku6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.170.148:59805/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-23 20:11:29,560 [I] Table x_portal_user already exists in database 'ranger'
2026-09-23 20:11:29,560 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskqUuku6/build/asan/bin/postgresql.jar:/tmp/dist-test-taskqUuku6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.170.148:59805/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-23 20:11:29,807 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-23 20:11:29,807 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskqUuku6/build/asan/bin/postgresql.jar:/tmp/dist-test-taskqUuku6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.170.148:59805/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-23 20:11:30,052 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskqUuku6/build/asan/bin/postgresql.jar:/tmp/dist-test-taskqUuku6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.170.148:59805/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-23 20:11:30,311 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskqUuku6/build/asan/bin/postgresql.jar:/tmp/dist-test-taskqUuku6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.170.148:59805/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-th4g.c.gcp-upstream.internal';"
2026-09-23 20:11:30,548 [I] CORE_DB_SCHEMA import status has been updated
2026-09-23 20:11:30,548 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskqUuku6/build/asan/bin/postgresql.jar:/tmp/dist-test-taskqUuku6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.170.148:59805/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-23 20:11:30,798 [I] DB_PATCHES have already been applied
I20260923 20:11:30.804118 1706 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194121577027-1706-0/minicluster-data/ranger-admin:/tmp/dist-test-taskqUuku6/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskqUuku6/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskqUuku6/build/asan/bin/postgresql.jar:/tmp/dist-test-taskqUuku6/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260923 20:11:30.804224 1706 mini_ranger.cc:193] Using host: 127.1.170.148
I20260923 20:11:30.819442 1706 mini_ranger.cc:227] Ranger admin URL: http://127.1.170.148:50211
Sep 23, 2026 8:11:31 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 23, 2026 8:11:31 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 23, 2026 8:11:31 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194121577027-1706-0/minicluster-data/ranger-admin/ews/webapp
Sep 23, 2026 8:11:31 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194121577027-1706-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 23, 2026 8:11:31 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194121577027-1706-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 23, 2026 8:11:31 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194121577027-1706-0/minicluster-data/ranger-admin/ews/webapp].
Sep 23, 2026 8:11:31 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260923 20:11:31.976919 1706 mini_ranger.cc:160] Time spent starting Ranger: real 8.841s user 0.003s sys 0.037s
Sep 23, 2026 8:11:52 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 23, 2026 8:11:52 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 23, 2026 8:11:52 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 23, 2026 8:11:52 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 23, 2026 8:11:52 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 23, 2026 8:11:52 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 23, 2026 8:11:52 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 23, 2026 8:11:52 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 23, 2026 8:11:52 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 23, 2026 8:11:52 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 23, 2026 8:11:52 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 23, 2026 8:11:52 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 23, 2026 8:11:52 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 23, 2026 8:11:52 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 23, 2026 8:11:52 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 23, 2026 8:11:52 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 23, 2026 8:11:52 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 23, 2026 8:11:52 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260923 20:11:53.592610 1706 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.1.170.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.1.170.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.1.170.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194121577027-1706-0/krb5kdc/rangerkms_127.1.170.148@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.1.170.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194121577027-1706-0/krb5kdc/rangerkms_127.1.170.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.1.170.148@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.1.170.148@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.1.170.148@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194121577027-1706-0/krb5kdc/HTTP_127.1.170.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.1.170.148@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194121577027-1706-0/krb5kdc/HTTP_127.1.170.148@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 23 20:11:53 dist-test-slave-th4g krb5kdc[13616](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194313, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260923 20:11:53.769021 1706 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260923 20:11:53.790591 1706 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260923 20:11:54.763208 1706 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260923 20:11:54.763334 1706 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194121577027-1706-0/minicluster-data/ranger-kms
I20260923 20:11:54.763402 1706 mini_ranger_kms.cc:227] Using postgres at 127.1.170.148:59805
2026-09-23 20:11:55,131 [I] DB FLAVOR :POSTGRES
2026-09-23 20:11:55,131 [I] --------- Verifying Ranger DB connection ---------
2026-09-23 20:11:55,131 [I] Checking connection
2026-09-23 20:11:55,131 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskqUuku6/build/asan/bin/postgresql.jar:/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194121577027-1706-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.170.148:59805/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-23 20:11:55,396 [I] connection success
2026-09-23 20:11:55,396 [I] --------- Verifying Ranger DB tables ---------
2026-09-23 20:11:55,396 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-23 20:11:55,396 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskqUuku6/build/asan/bin/postgresql.jar:/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194121577027-1706-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.170.148:59805/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-23 20:11:55,647 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-23 20:11:55,647 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-23 20:11:55,647 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-23 20:11:55,647 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskqUuku6/build/asan/bin/postgresql.jar:/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194121577027-1706-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.170.148:59805/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194121577027-1706-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-23 20:11:55,950 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260923 20:11:56.503141 1706 mini_ranger_kms.cc:310] Created kms service
I20260923 20:11:56.708230 1706 mini_ranger_kms.cc:326] Created kudu user
I20260923 20:11:56.752964 1706 mini_ranger_kms.cc:343] Created rangerkms user
Sep 23 20:11:56 dist-test-slave-th4g krb5kdc[13616](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194313, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.1.170.148@KRBTEST.COM
I20260923 20:11:57.163981 1706 mini_ranger_kms.cc:384] Added ranger policy
I20260923 20:11:57.164281 1706 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194121577027-1706-0/minicluster-data/ranger-kms:/tmp/dist-test-taskqUuku6/build/asan/bin/postgresql.jar:/tmp/dist-test-taskqUuku6/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskqUuku6/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260923 20:11:57.164332 1706 mini_ranger_kms.cc:240] Using host: 127.1.170.148
I20260923 20:11:57.184273 1706 mini_ranger_kms.cc:276] Ranger KMS PID: 14009
I20260923 20:11:57.184407 1706 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.1.170.148:39063
Sep 23, 2026 8:11:58 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 23, 2026 8:11:58 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 23, 2026 8:11:58 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194121577027-1706-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 23, 2026 8:11:58 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194121577027-1706-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 23, 2026 8:11:58 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194121577027-1706-0/minicluster-data/ranger-kms/ews/webapp].
Sep 23, 2026 8:11:58 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260923 20:11:59.478024 1706 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.709s user 0.076s sys 0.305s
I20260923 20:11:59.478276 1706 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260923 20:11:59.478446 1706 mini_ranger_kms.cc:401] 127.1.170.148:39063/kms/v1/keys
Sep 23 20:11:59 dist-test-slave-th4g krb5kdc[13616](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790194319, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.1.170.148@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 23 20:12:00 dist-test-slave-th4g krb5kdc[13616](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194319, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.1.170.148@KRBTEST.COM for HTTP/127.1.170.148@KRBTEST.COM
Sep 23 20:12:00 dist-test-slave-th4g krb5kdc[13616](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194319, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.1.170.148@KRBTEST.COM for HTTP/127.1.170.148@KRBTEST.COM
Sep 23 20:12:02 dist-test-slave-th4g krb5kdc[13616](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194322, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.1.170.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.170.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.170.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194121577027-1706-0/krb5kdc/kudu_127.1.170.190.keytab.
Entry for principal kudu/127.1.170.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194121577027-1706-0/krb5kdc/kudu_127.1.170.190.keytab.
I20260923 20:12:02.686530 1706 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqUuku6/build/asan/bin/kudu
/tmp/dist-test-taskqUuku6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194121577027-1706-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194121577027-1706-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194121577027-1706-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194121577027-1706-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.170.190:43227
--webserver_interface=127.1.170.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194121577027-1706-0/krb5kdc/kudu_127.1.170.190.keytab
--principal=kudu/127.1.170.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.170.148:34035
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.170.190:43227
--ranger_config_path=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194121577027-1706-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.1.170.148:39063/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194121577027-1706-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194121577027-1706-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194121577027-1706-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:12:03.017009 14060 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:12:03.017304 14060 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:12:03.017416 14060 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:12:03.025013 14060 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 20:12:03.025135 14060 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:12:03.025180 14060 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:12:03.025235 14060 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 20:12:03.025283 14060 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 20:12:03.035634 14060 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.170.148:34035
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194121577027-1706-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194121577027-1706-0/minicluster-data/master-0/wal
--ranger_kms_url=127.1.170.148:39063/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.1.170.190:43227
--ranger_config_path=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194121577027-1706-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194121577027-1706-0/krb5kdc/kudu_127.1.170.190.keytab
--principal=kudu/127.1.170.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.170.190:43227
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194121577027-1706-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.170.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194121577027-1706-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:12:03.036880 14060 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:12:03.038223 14060 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:12:03.050189 14065 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:12:03.050208 14068 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:12:03.050238 14066 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:12:03.050892 14060 server_base.cc:1034] running on GCE node
I20260923 20:12:03.051683 14060 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:12:03.053053 14060 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:12:03.054245 14060 hybrid_clock.cc:648] HybridClock initialized: now 1790194323054204 us; error 34 us; skew 500 ppm
I20260923 20:12:03.054544 14060 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:12:03 dist-test-slave-th4g krb5kdc[13616](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194323, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.170.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:12:03.061636 14060 init.cc:377] Logged in from keytab as kudu/127.1.170.190@KRBTEST.COM (short username kudu)
I20260923 20:12:03.063333 14060 webserver.cc:460] Webserver started at http://127.1.170.190:37477/ using document root <none> and password file <none>
I20260923 20:12:03.064066 14060 fs_manager.cc:359] Metadata directory not provided
I20260923 20:12:03.064199 14060 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:12:03.064531 14060 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 23 20:12:03 dist-test-slave-th4g krb5kdc[13616](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194323, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.170.190@KRBTEST.COM for HTTP/127.1.170.148@KRBTEST.COM
I20260923 20:12:03.124362 14060 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194121577027-1706-0/minicluster-data/master-0/data/instance:
uuid: "528212d773b24273adebf10a109a5abe"
format_stamp: "Formatted at 2026-09-23 20:12:03 on dist-test-slave-th4g"
server_key: "644f02c5d8228a4e0406124d68f025bb"
server_key_iv: "113a11d15ecbff116130b0f6319b21fc"
server_key_version: "kuduclusterkey@0"
I20260923 20:12:03.125530 14060 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194121577027-1706-0/minicluster-data/master-0/wal/instance:
uuid: "528212d773b24273adebf10a109a5abe"
format_stamp: "Formatted at 2026-09-23 20:12:03 on dist-test-slave-th4g"
server_key: "644f02c5d8228a4e0406124d68f025bb"
server_key_iv: "113a11d15ecbff116130b0f6319b21fc"
server_key_version: "kuduclusterkey@0"
I20260923 20:12:03.132505 14060 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.001s sys 0.004s
I20260923 20:12:03.153576 14075 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:12:03.155225 14060 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260923 20:12:03.155459 14060 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194121577027-1706-0/minicluster-data/master-0/data,/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194121577027-1706-0/minicluster-data/master-0/wal
uuid: "528212d773b24273adebf10a109a5abe"
format_stamp: "Formatted at 2026-09-23 20:12:03 on dist-test-slave-th4g"
server_key: "644f02c5d8228a4e0406124d68f025bb"
server_key_iv: "113a11d15ecbff116130b0f6319b21fc"
server_key_version: "kuduclusterkey@0"
I20260923 20:12:03.155704 14060 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194121577027-1706-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194121577027-1706-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194121577027-1706-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:12:03.173314 14060 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:12:03.177736 14060 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:12:03.178100 14060 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:12:03.197083 14060 rpc_server.cc:307] RPC server started. Bound to: 127.1.170.190:43227
I20260923 20:12:03.197154 14126 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.170.190:43227 every 8 connection(s)
I20260923 20:12:03.198681 14060 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194121577027-1706-0/minicluster-data/master-0/data/info.pb
I20260923 20:12:03.200426 1706 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqUuku6/build/asan/bin/kudu as pid 14060
I20260923 20:12:03.200629 1706 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194121577027-1706-0/minicluster-data/master-0/wal/instance
I20260923 20:12:03.201264 1706 external_mini_cluster.cc:1428] Setting key 4e6528eff208a0642e2c386742da0f91
I20260923 20:12:03.202735 14127 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 23 20:12:03 dist-test-slave-th4g krb5kdc[13616](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194322, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.170.190@KRBTEST.COM
I20260923 20:12:03.214275 14127 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 528212d773b24273adebf10a109a5abe: Bootstrap starting.
I20260923 20:12:03.217640 14127 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 528212d773b24273adebf10a109a5abe: Neither blocks nor log segments found. Creating new log.
I20260923 20:12:03.218755 14127 log.cc:824] T 00000000000000000000000000000000 P 528212d773b24273adebf10a109a5abe: Log is configured to *not* fsync() on all Append() calls
I20260923 20:12:03.221709 14127 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 528212d773b24273adebf10a109a5abe: No bootstrap required, opened a new log
I20260923 20:12:03.222292 14130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:12:03.203210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35516 (local address 127.1.170.190:43227)
0923 20:12:03.203970 (+ 760us) server_negotiation.cc:206] Beginning negotiation
0923 20:12:03.203982 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0923 20:12:03.204019 (+ 37us) server_negotiation.cc:407] Connection header received
0923 20:12:03.205540 (+ 1521us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:12:03.205560 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:12:03.206014 (+ 454us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:12:03.206393 (+ 379us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:12:03.207726 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:12:03.209082 (+ 1356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:12:03.210462 (+ 1380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:12:03.211193 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:12:03.211417 (+ 224us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:12:03.215232 (+ 3815us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:12:03.215239 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:12:03.215253 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:12:03.215288 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:12:03.218314 (+ 3026us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:12:03.218978 (+ 664us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:12:03.218985 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:12:03.218990 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:12:03.219141 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:12:03.219543 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:12:03.219547 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:12:03.219549 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:12:03.220152 (+ 603us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:12:03.220444 (+ 292us) server_negotiation.cc:1035] Waiting for connection context
0923 20:12:03.220879 (+ 435us) server_negotiation.cc:299] Negotiation successful
0923 20:12:03.221267 (+ 388us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":568,"thread_start_us":375,"threads_started":1}
I20260923 20:12:03.229431 14127 raft_consensus.cc:348] T 00000000000000000000000000000000 P 528212d773b24273adebf10a109a5abe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "528212d773b24273adebf10a109a5abe" member_type: VOTER last_known_addr { host: "127.1.170.190" port: 43227 } }
I20260923 20:12:03.229799 14127 raft_consensus.cc:374] T 00000000000000000000000000000000 P 528212d773b24273adebf10a109a5abe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:12:03.229920 14127 raft_consensus.cc:729] T 00000000000000000000000000000000 P 528212d773b24273adebf10a109a5abe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 528212d773b24273adebf10a109a5abe, State: Initialized, Role: FOLLOWER
I20260923 20:12:03.230445 14127 consensus_queue.cc:260] T 00000000000000000000000000000000 P 528212d773b24273adebf10a109a5abe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "528212d773b24273adebf10a109a5abe" member_type: VOTER last_known_addr { host: "127.1.170.190" port: 43227 } }
I20260923 20:12:03.230698 14127 raft_consensus.cc:388] T 00000000000000000000000000000000 P 528212d773b24273adebf10a109a5abe [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 20:12:03.230858 14127 raft_consensus.cc:482] T 00000000000000000000000000000000 P 528212d773b24273adebf10a109a5abe [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 20:12:03.231060 14127 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 528212d773b24273adebf10a109a5abe [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:12:03.233186 14127 raft_consensus.cc:504] T 00000000000000000000000000000000 P 528212d773b24273adebf10a109a5abe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "528212d773b24273adebf10a109a5abe" member_type: VOTER last_known_addr { host: "127.1.170.190" port: 43227 } }
I20260923 20:12:03.233675 14127 leader_election.cc:302] T 00000000000000000000000000000000 P 528212d773b24273adebf10a109a5abe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 528212d773b24273adebf10a109a5abe; no voters:
I20260923 20:12:03.234342 14127 leader_election.cc:288] T 00000000000000000000000000000000 P 528212d773b24273adebf10a109a5abe [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 20:12:03.234519 14132 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 528212d773b24273adebf10a109a5abe [term 1 FOLLOWER]: Leader election won for term 1
I20260923 20:12:03.235183 14132 raft_consensus.cc:686] T 00000000000000000000000000000000 P 528212d773b24273adebf10a109a5abe [term 1 LEADER]: Becoming Leader. State: Replica: 528212d773b24273adebf10a109a5abe, State: Running, Role: LEADER
I20260923 20:12:03.235697 14132 consensus_queue.cc:237] T 00000000000000000000000000000000 P 528212d773b24273adebf10a109a5abe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "528212d773b24273adebf10a109a5abe" member_type: VOTER last_known_addr { host: "127.1.170.190" port: 43227 } }
I20260923 20:12:03.236582 14127 sys_catalog.cc:564] T 00000000000000000000000000000000 P 528212d773b24273adebf10a109a5abe [sys.catalog]: configured and running, proceeding with master startup.
I20260923 20:12:03.238550 14134 sys_catalog.cc:455] T 00000000000000000000000000000000 P 528212d773b24273adebf10a109a5abe [sys.catalog]: SysCatalogTable state changed. Reason: New leader 528212d773b24273adebf10a109a5abe. Latest consensus state: current_term: 1 leader_uuid: "528212d773b24273adebf10a109a5abe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "528212d773b24273adebf10a109a5abe" member_type: VOTER last_known_addr { host: "127.1.170.190" port: 43227 } } }
I20260923 20:12:03.238576 14133 sys_catalog.cc:455] T 00000000000000000000000000000000 P 528212d773b24273adebf10a109a5abe [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "528212d773b24273adebf10a109a5abe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "528212d773b24273adebf10a109a5abe" member_type: VOTER last_known_addr { host: "127.1.170.190" port: 43227 } } }
I20260923 20:12:03.238983 14134 sys_catalog.cc:458] T 00000000000000000000000000000000 P 528212d773b24273adebf10a109a5abe [sys.catalog]: This master's current role is: LEADER
I20260923 20:12:03.238983 14133 sys_catalog.cc:458] T 00000000000000000000000000000000 P 528212d773b24273adebf10a109a5abe [sys.catalog]: This master's current role is: LEADER
I20260923 20:12:03.240823 14138 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 20:12:03.246213 14138 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 20:12:03.250038 14127 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194121577027-1706-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260923 20:12:03.262745 14138 catalog_manager.cc:1347] Generated new cluster ID: 59f84a8184324d128eed7530192a26c4
I20260923 20:12:03.262948 14138 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 20:12:03.278024 14138 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 20:12:03.279446 14138 catalog_manager.cc:1504] Loading token signing keys...
I20260923 20:12:03.287670 14138 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 528212d773b24273adebf10a109a5abe: Generated new TSK 0
I20260923 20:12:03.288471 14138 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.170.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.170.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.170.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194121577027-1706-0/krb5kdc/kudu_127.1.170.129.keytab.
Entry for principal kudu/127.1.170.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194121577027-1706-0/krb5kdc/kudu_127.1.170.129.keytab.
I20260923 20:12:04.058658 1706 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqUuku6/build/asan/bin/kudu
/tmp/dist-test-taskqUuku6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194121577027-1706-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194121577027-1706-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194121577027-1706-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194121577027-1706-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.1.170.148:39063/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.170.129:0
--local_ip_for_outbound_sockets=127.1.170.129
--webserver_interface=127.1.170.129
--webserver_port=0
--tserver_master_addrs=127.1.170.190:43227
--keytab_file=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194121577027-1706-0/krb5kdc/kudu_127.1.170.129.keytab
--principal=kudu/127.1.170.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.170.148:34035
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194121577027-1706-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194121577027-1706-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194121577027-1706-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 23 20:12:04 dist-test-slave-th4g krb5kdc[13616](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790194324, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.170.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260923 20:12:04.515134 14172 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:12:04.515627 14172 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:12:04.515817 14172 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:12:04.527621 14172 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:12:04.527838 14172 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:12:04.528054 14172 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.170.129
I20260923 20:12:04.541276 14172 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.170.148:34035
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194121577027-1706-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194121577027-1706-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.1.170.148:39063/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194121577027-1706-0/krb5kdc/kudu_127.1.170.129.keytab
--principal=kudu/127.1.170.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.170.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194121577027-1706-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.170.129
--webserver_port=0
--tserver_master_addrs=127.1.170.190:43227
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.170.129
--log_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194121577027-1706-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:12:04.542763 14172 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:12:04.544130 14172 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:12:04.558320 14183 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:12:04.558616 14182 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:12:04.559010 14185 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:12:04.560010 14172 server_base.cc:1034] running on GCE node
I20260923 20:12:04.560803 14172 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:12:04.562091 14172 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:12:04.569480 14172 hybrid_clock.cc:648] HybridClock initialized: now 1790194324569356 us; error 93 us; skew 500 ppm
I20260923 20:12:04.570003 14172 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:12:04 dist-test-slave-th4g krb5kdc[13616](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194324, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.170.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:12:04.574540 14172 init.cc:377] Logged in from keytab as kudu/127.1.170.129@KRBTEST.COM (short username kudu)
I20260923 20:12:04.587379 14172 webserver.cc:460] Webserver started at http://127.1.170.129:39207/ using document root <none> and password file <none>
I20260923 20:12:04.588398 14172 fs_manager.cc:359] Metadata directory not provided
I20260923 20:12:04.588632 14172 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:12:04.589078 14172 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 23 20:12:04 dist-test-slave-th4g krb5kdc[13616](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194324, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.170.129@KRBTEST.COM for HTTP/127.1.170.148@KRBTEST.COM
I20260923 20:12:04.620249 14172 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194121577027-1706-0/minicluster-data/ts-0/data/instance:
uuid: "711ac19c204c49e5842f923bebce66cc"
format_stamp: "Formatted at 2026-09-23 20:12:04 on dist-test-slave-th4g"
server_key: "f34add8cd89b65ca991f0eccd3468edb"
server_key_iv: "ed6bddd757467a2bdce1f7300dd41ac4"
server_key_version: "kuduclusterkey@0"
I20260923 20:12:04.621419 14172 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194121577027-1706-0/minicluster-data/ts-0/wal/instance:
uuid: "711ac19c204c49e5842f923bebce66cc"
format_stamp: "Formatted at 2026-09-23 20:12:04 on dist-test-slave-th4g"
server_key: "f34add8cd89b65ca991f0eccd3468edb"
server_key_iv: "ed6bddd757467a2bdce1f7300dd41ac4"
server_key_version: "kuduclusterkey@0"
I20260923 20:12:04.630620 14172 fs_manager.cc:691] Time spent creating directory manager: real 0.009s user 0.005s sys 0.004s
I20260923 20:12:04.659855 14192 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:12:04.662009 14172 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.006s sys 0.000s
I20260923 20:12:04.662212 14172 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194121577027-1706-0/minicluster-data/ts-0/data,/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194121577027-1706-0/minicluster-data/ts-0/wal
uuid: "711ac19c204c49e5842f923bebce66cc"
format_stamp: "Formatted at 2026-09-23 20:12:04 on dist-test-slave-th4g"
server_key: "f34add8cd89b65ca991f0eccd3468edb"
server_key_iv: "ed6bddd757467a2bdce1f7300dd41ac4"
server_key_version: "kuduclusterkey@0"
I20260923 20:12:04.662464 14172 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194121577027-1706-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194121577027-1706-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194121577027-1706-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:12:04.720913 14172 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:12:04.726703 14172 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
Sep 23 20:12:04 dist-test-slave-th4g krb5kdc[13616](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194324, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.170.190@KRBTEST.COM for HTTP/127.1.170.148@KRBTEST.COM
I20260923 20:12:04.727769 14172 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:12:04.735445 14172 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 20:12:04.743906 14172 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 20:12:04.744100 14172 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260923 20:12:04.744303 14172 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 20:12:04.744438 14172 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 20:12:05.126287 14172 rpc_server.cc:307] RPC server started. Bound to: 127.1.170.129:44269
I20260923 20:12:05.126297 14304 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.170.129:44269 every 8 connection(s)
I20260923 20:12:05.127828 14172 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194121577027-1706-0/minicluster-data/ts-0/data/info.pb
I20260923 20:12:05.134198 1706 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqUuku6/build/asan/bin/kudu as pid 14172
I20260923 20:12:05.134395 1706 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194121577027-1706-0/minicluster-data/ts-0/wal/instance
I20260923 20:12:05.135254 1706 external_mini_cluster.cc:1428] Setting key d960f7a6f2b14fe0b33524e6f96ca4f1
Sep 23 20:12:05 dist-test-slave-th4g krb5kdc[13616](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194324, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.170.129@KRBTEST.COM for kudu/127.1.170.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.1.170.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.170.130@KRBTEST.COM" created.
I20260923 20:12:05.188411 14308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:12:05.132677 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.170.129:35475 (local address 127.1.170.190:43227)
0923 20:12:05.168034 (+ 35357us) server_negotiation.cc:206] Beginning negotiation
0923 20:12:05.168041 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:12:05.168069 (+ 28us) server_negotiation.cc:407] Connection header received
0923 20:12:05.168153 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:12:05.168158 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:12:05.168269 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:12:05.168413 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:12:05.170859 (+ 2446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:12:05.172048 (+ 1189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:12:05.174683 (+ 2635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:12:05.175321 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:12:05.175429 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:12:05.180252 (+ 4823us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:12:05.180259 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:12:05.180264 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:12:05.180321 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:12:05.184076 (+ 3755us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:12:05.185189 (+ 1113us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:12:05.185196 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:12:05.185199 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:12:05.185421 (+ 222us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:12:05.186111 (+ 690us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:12:05.186116 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:12:05.186120 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:12:05.186503 (+ 383us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:12:05.186669 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0923 20:12:05.187757 (+ 1088us) server_negotiation.cc:299] Negotiation successful
0923 20:12:05.188015 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35182,"thread_start_us":35099,"threads_started":1}
I20260923 20:12:05.190893 14307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:12:05.131987 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.190:43227 (local address 127.1.170.129:35475)
0923 20:12:05.162002 (+ 30015us) negotiation.cc:107] Waiting for socket to connect
0923 20:12:05.162063 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0923 20:12:05.164494 (+ 2431us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:12:05.168804 (+ 4310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:12:05.168819 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:12:05.169576 (+ 757us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:12:05.170521 (+ 945us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:12:05.170551 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:12:05.172260 (+ 1709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:12:05.172266 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:12:05.174466 (+ 2200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:12:05.174496 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:12:05.175833 (+ 1337us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:12:05.175841 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:12:05.176132 (+ 291us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:12:05.176974 (+ 842us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:12:05.177006 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:12:05.179926 (+ 2920us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:12:05.184284 (+ 4358us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:12:05.184296 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:12:05.184316 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:12:05.185009 (+ 693us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:12:05.185614 (+ 605us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:12:05.185620 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:12:05.185624 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:12:05.185971 (+ 347us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:12:05.186752 (+ 781us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:12:05.186764 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:12:05.187258 (+ 494us) client_negotiation.cc:769] Sending connection context
0923 20:12:05.188731 (+ 1473us) client_negotiation.cc:240] Negotiation successful
0923 20:12:05.189317 (+ 586us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29645,"thread_start_us":28443,"threads_started":1}
I20260923 20:12:05.193535 14305 heartbeater.cc:347] Connected to a master server at 127.1.170.190:43227
I20260923 20:12:05.193923 14305 heartbeater.cc:464] Registering TS with master...
I20260923 20:12:05.194840 14305 heartbeater.cc:511] Master 127.1.170.190:43227 requested a full tablet report, sending...
I20260923 20:12:05.197628 14092 ts_manager.cc:194] Registered new tserver with Master: 711ac19c204c49e5842f923bebce66cc (127.1.170.129:44269)
I20260923 20:12:05.199954 14092 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.170.129@KRBTEST.COM'} at 127.1.170.129:35475
Sep 23 20:12:05 dist-test-slave-th4g krb5kdc[13616](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194324, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.170.190@KRBTEST.COM for HTTP/127.1.170.148@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.170.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194121577027-1706-0/krb5kdc/kudu_127.1.170.130.keytab.
Entry for principal kudu/127.1.170.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194121577027-1706-0/krb5kdc/kudu_127.1.170.130.keytab.
W20260923 20:12:05.239089 1706 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.1.170.130: real 0.101s user 0.004s sys 0.013s
I20260923 20:12:05.240059 1706 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqUuku6/build/asan/bin/kudu
/tmp/dist-test-taskqUuku6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194121577027-1706-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194121577027-1706-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194121577027-1706-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194121577027-1706-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.1.170.148:39063/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.170.130:0
--local_ip_for_outbound_sockets=127.1.170.130
--webserver_interface=127.1.170.130
--webserver_port=0
--tserver_master_addrs=127.1.170.190:43227
--keytab_file=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194121577027-1706-0/krb5kdc/kudu_127.1.170.130.keytab
--principal=kudu/127.1.170.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.170.148:34035
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194121577027-1706-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194121577027-1706-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194121577027-1706-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:12:05.685344 14312 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:12:05.685711 14312 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:12:05.685914 14312 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:12:05.693748 14312 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:12:05.693938 14312 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:12:05.694116 14312 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.170.130
I20260923 20:12:05.704303 14312 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.170.148:34035
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194121577027-1706-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194121577027-1706-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.1.170.148:39063/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194121577027-1706-0/krb5kdc/kudu_127.1.170.130.keytab
--principal=kudu/127.1.170.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.170.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194121577027-1706-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.170.130
--webserver_port=0
--tserver_master_addrs=127.1.170.190:43227
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.170.130
--log_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194121577027-1706-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:12:05.705777 14312 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:12:05.707098 14312 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:12:05.720649 14325 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:12:05.720696 14328 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:12:05.720610 14326 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:12:05.721590 14312 server_base.cc:1034] running on GCE node
I20260923 20:12:05.722308 14312 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:12:05.723789 14312 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:12:05.725044 14312 hybrid_clock.cc:648] HybridClock initialized: now 1790194325724962 us; error 60 us; skew 500 ppm
I20260923 20:12:05.725406 14312 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:12:05 dist-test-slave-th4g krb5kdc[13616](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194325, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.170.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:12:05.728766 14312 init.cc:377] Logged in from keytab as kudu/127.1.170.130@KRBTEST.COM (short username kudu)
I20260923 20:12:05.730399 14312 webserver.cc:460] Webserver started at http://127.1.170.130:36041/ using document root <none> and password file <none>
I20260923 20:12:05.731055 14312 fs_manager.cc:359] Metadata directory not provided
I20260923 20:12:05.731166 14312 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:12:05.731419 14312 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 23 20:12:05 dist-test-slave-th4g krb5kdc[13616](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194325, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.170.130@KRBTEST.COM for HTTP/127.1.170.148@KRBTEST.COM
I20260923 20:12:05.749785 14312 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194121577027-1706-0/minicluster-data/ts-1/data/instance:
uuid: "0bcb955ca73c4fdd8b6b2307210c43ac"
format_stamp: "Formatted at 2026-09-23 20:12:05 on dist-test-slave-th4g"
server_key: "5541006d2c0f4667379ea0a331f17d64"
server_key_iv: "f224528856ed3e5b0b05580bcd1ed75a"
server_key_version: "kuduclusterkey@0"
I20260923 20:12:05.750821 14312 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194121577027-1706-0/minicluster-data/ts-1/wal/instance:
uuid: "0bcb955ca73c4fdd8b6b2307210c43ac"
format_stamp: "Formatted at 2026-09-23 20:12:05 on dist-test-slave-th4g"
server_key: "5541006d2c0f4667379ea0a331f17d64"
server_key_iv: "f224528856ed3e5b0b05580bcd1ed75a"
server_key_version: "kuduclusterkey@0"
I20260923 20:12:05.759943 14312 fs_manager.cc:691] Time spent creating directory manager: real 0.009s user 0.007s sys 0.001s
I20260923 20:12:05.778577 14335 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:12:05.780143 14312 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260923 20:12:05.780319 14312 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194121577027-1706-0/minicluster-data/ts-1/data,/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194121577027-1706-0/minicluster-data/ts-1/wal
uuid: "0bcb955ca73c4fdd8b6b2307210c43ac"
format_stamp: "Formatted at 2026-09-23 20:12:05 on dist-test-slave-th4g"
server_key: "5541006d2c0f4667379ea0a331f17d64"
server_key_iv: "f224528856ed3e5b0b05580bcd1ed75a"
server_key_version: "kuduclusterkey@0"
I20260923 20:12:05.780567 14312 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194121577027-1706-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194121577027-1706-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194121577027-1706-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:12:05.817400 14312 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:12:05.821810 14312 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:12:05.822187 14312 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:12:05.823290 14312 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 20:12:05.824704 14312 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 20:12:05.824780 14312 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 20:12:05.824936 14312 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 20:12:05.824986 14312 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 20:12:05.868965 14312 rpc_server.cc:307] RPC server started. Bound to: 127.1.170.130:45645
I20260923 20:12:05.868976 14447 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.170.130:45645 every 8 connection(s)
I20260923 20:12:05.870390 14312 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194121577027-1706-0/minicluster-data/ts-1/data/info.pb
I20260923 20:12:05.872745 1706 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqUuku6/build/asan/bin/kudu as pid 14312
I20260923 20:12:05.872910 1706 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194121577027-1706-0/minicluster-data/ts-1/wal/instance
I20260923 20:12:05.873481 1706 external_mini_cluster.cc:1428] Setting key 7f6b2a4706256c4d1db48a891bdb574e
Sep 23 20:12:05 dist-test-slave-th4g krb5kdc[13616](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194325, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.170.130@KRBTEST.COM for kudu/127.1.170.190@KRBTEST.COM
I20260923 20:12:05.906731 14450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:12:05.873072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.170.130:57675 (local address 127.1.170.190:43227)
0923 20:12:05.882123 (+ 9051us) server_negotiation.cc:206] Beginning negotiation
0923 20:12:05.882130 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:12:05.883051 (+ 921us) server_negotiation.cc:407] Connection header received
0923 20:12:05.885973 (+ 2922us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:12:05.885986 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:12:05.886115 (+ 129us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:12:05.886248 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:12:05.888296 (+ 2048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:12:05.889506 (+ 1210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:12:05.892098 (+ 2592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:12:05.892985 (+ 887us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:12:05.893197 (+ 212us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:12:05.897699 (+ 4502us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:12:05.897709 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:12:05.897715 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:12:05.897768 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:12:05.901731 (+ 3963us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:12:05.902913 (+ 1182us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:12:05.902922 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:12:05.902928 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:12:05.903102 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:12:05.903849 (+ 747us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:12:05.903857 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:12:05.903863 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:12:05.904286 (+ 423us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:12:05.904459 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0923 20:12:05.905846 (+ 1387us) server_negotiation.cc:299] Negotiation successful
0923 20:12:05.906196 (+ 350us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8893,"thread_start_us":8741,"threads_started":1}
WARNING: no policy specified for kudu/127.1.170.131@KRBTEST.COM; defaulting to no policy
I20260923 20:12:05.908816 14451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:12:05.873922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.190:43227 (local address 127.1.170.130:57675)
0923 20:12:05.882807 (+ 8885us) negotiation.cc:107] Waiting for socket to connect
0923 20:12:05.882880 (+ 73us) client_negotiation.cc:174] Beginning negotiation
0923 20:12:05.885495 (+ 2615us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:12:05.886571 (+ 1076us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:12:05.886584 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:12:05.887223 (+ 639us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:12:05.888059 (+ 836us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:12:05.888084 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:12:05.889730 (+ 1646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:12:05.889736 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:12:05.891893 (+ 2157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:12:05.891911 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:12:05.893274 (+ 1363us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:12:05.893283 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:12:05.893600 (+ 317us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:12:05.894497 (+ 897us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:12:05.894528 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:12:05.897364 (+ 2836us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:12:05.902007 (+ 4643us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:12:05.902019 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:12:05.902041 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:12:05.902731 (+ 690us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:12:05.903235 (+ 504us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:12:05.903240 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:12:05.903244 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:12:05.903531 (+ 287us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:12:05.904511 (+ 980us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:12:05.904525 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:12:05.905208 (+ 683us) client_negotiation.cc:769] Sending connection context
0923 20:12:05.905656 (+ 448us) client_negotiation.cc:240] Negotiation successful
0923 20:12:05.906509 (+ 853us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8530,"thread_start_us":8239,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.170.131@KRBTEST.COM" created.
I20260923 20:12:05.911491 14448 heartbeater.cc:347] Connected to a master server at 127.1.170.190:43227
I20260923 20:12:05.911939 14448 heartbeater.cc:464] Registering TS with master...
I20260923 20:12:05.912930 14448 heartbeater.cc:511] Master 127.1.170.190:43227 requested a full tablet report, sending...
I20260923 20:12:05.914966 14092 ts_manager.cc:194] Registered new tserver with Master: 0bcb955ca73c4fdd8b6b2307210c43ac (127.1.170.130:45645)
I20260923 20:12:05.915927 14092 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.170.130@KRBTEST.COM'} at 127.1.170.130:57675
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.170.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194121577027-1706-0/krb5kdc/kudu_127.1.170.131.keytab.
Entry for principal kudu/127.1.170.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194121577027-1706-0/krb5kdc/kudu_127.1.170.131.keytab.
I20260923 20:12:05.947829 1706 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqUuku6/build/asan/bin/kudu
/tmp/dist-test-taskqUuku6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194121577027-1706-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194121577027-1706-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194121577027-1706-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194121577027-1706-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.1.170.148:39063/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.170.131:0
--local_ip_for_outbound_sockets=127.1.170.131
--webserver_interface=127.1.170.131
--webserver_port=0
--tserver_master_addrs=127.1.170.190:43227
--keytab_file=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194121577027-1706-0/krb5kdc/kudu_127.1.170.131.keytab
--principal=kudu/127.1.170.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.170.148:34035
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194121577027-1706-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194121577027-1706-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194121577027-1706-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260923 20:12:06.203832 14305 heartbeater.cc:503] Master 127.1.170.190:43227 was elected leader, sending a full tablet report...
W20260923 20:12:06.311120 14454 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:12:06.311486 14454 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:12:06.311653 14454 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:12:06.319074 14454 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:12:06.319245 14454 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:12:06.319435 14454 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.170.131
I20260923 20:12:06.329238 14454 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.170.148:34035
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194121577027-1706-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194121577027-1706-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.1.170.148:39063/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194121577027-1706-0/krb5kdc/kudu_127.1.170.131.keytab
--principal=kudu/127.1.170.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.170.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194121577027-1706-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.170.131
--webserver_port=0
--tserver_master_addrs=127.1.170.190:43227
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.170.131
--log_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194121577027-1706-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:12:06.330606 14454 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:12:06.331908 14454 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:12:06.344846 14459 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:12:06.344955 14460 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:12:06.345073 14462 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:12:06.346139 14454 server_base.cc:1034] running on GCE node
I20260923 20:12:06.346907 14454 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:12:06.347913 14454 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:12:06.349150 14454 hybrid_clock.cc:648] HybridClock initialized: now 1790194326349091 us; error 51 us; skew 500 ppm
I20260923 20:12:06.349566 14454 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:12:06 dist-test-slave-th4g krb5kdc[13616](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194326, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.170.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:12:06.353550 14454 init.cc:377] Logged in from keytab as kudu/127.1.170.131@KRBTEST.COM (short username kudu)
I20260923 20:12:06.355235 14454 webserver.cc:460] Webserver started at http://127.1.170.131:46865/ using document root <none> and password file <none>
I20260923 20:12:06.356037 14454 fs_manager.cc:359] Metadata directory not provided
I20260923 20:12:06.356150 14454 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:12:06.356403 14454 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 23 20:12:06 dist-test-slave-th4g krb5kdc[13616](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194326, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.170.131@KRBTEST.COM for HTTP/127.1.170.148@KRBTEST.COM
I20260923 20:12:06.372038 14454 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194121577027-1706-0/minicluster-data/ts-2/data/instance:
uuid: "ec85741e5ec64d42a20e3c914e696a87"
format_stamp: "Formatted at 2026-09-23 20:12:06 on dist-test-slave-th4g"
server_key: "041b42b62f06df43bbd36f9495fa06a0"
server_key_iv: "970cf46e1a7f15a4e417525b5e649400"
server_key_version: "kuduclusterkey@0"
I20260923 20:12:06.373029 14454 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194121577027-1706-0/minicluster-data/ts-2/wal/instance:
uuid: "ec85741e5ec64d42a20e3c914e696a87"
format_stamp: "Formatted at 2026-09-23 20:12:06 on dist-test-slave-th4g"
server_key: "041b42b62f06df43bbd36f9495fa06a0"
server_key_iv: "970cf46e1a7f15a4e417525b5e649400"
server_key_version: "kuduclusterkey@0"
I20260923 20:12:06.378786 14454 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260923 20:12:06.395646 14469 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:12:06.397190 14454 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260923 20:12:06.397403 14454 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194121577027-1706-0/minicluster-data/ts-2/data,/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194121577027-1706-0/minicluster-data/ts-2/wal
uuid: "ec85741e5ec64d42a20e3c914e696a87"
format_stamp: "Formatted at 2026-09-23 20:12:06 on dist-test-slave-th4g"
server_key: "041b42b62f06df43bbd36f9495fa06a0"
server_key_iv: "970cf46e1a7f15a4e417525b5e649400"
server_key_version: "kuduclusterkey@0"
I20260923 20:12:06.397634 14454 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194121577027-1706-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194121577027-1706-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194121577027-1706-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:12:06.413791 14454 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:12:06.418128 14454 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:12:06.418416 14454 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:12:06.419438 14454 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 20:12:06.420833 14454 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 20:12:06.420938 14454 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 20:12:06.421096 14454 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 20:12:06.421185 14454 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 20:12:06.457182 14454 rpc_server.cc:307] RPC server started. Bound to: 127.1.170.131:34661
I20260923 20:12:06.457186 14581 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.170.131:34661 every 8 connection(s)
I20260923 20:12:06.458581 14454 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194121577027-1706-0/minicluster-data/ts-2/data/info.pb
I20260923 20:12:06.463881 1706 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqUuku6/build/asan/bin/kudu as pid 14454
I20260923 20:12:06.464071 1706 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194121577027-1706-0/minicluster-data/ts-2/wal/instance
I20260923 20:12:06.464486 1706 external_mini_cluster.cc:1428] Setting key 2e31689c052cf56991f945bebfd02c8a
Sep 23 20:12:06 dist-test-slave-th4g krb5kdc[13616](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194326, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.170.131@KRBTEST.COM for kudu/127.1.170.190@KRBTEST.COM
I20260923 20:12:06.481511 14584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:12:06.461375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.170.131:58755 (local address 127.1.170.190:43227)
0923 20:12:06.461924 (+ 549us) server_negotiation.cc:206] Beginning negotiation
0923 20:12:06.461930 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:12:06.462952 (+ 1022us) server_negotiation.cc:407] Connection header received
0923 20:12:06.465338 (+ 2386us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:12:06.465343 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:12:06.465452 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:12:06.465576 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:12:06.467317 (+ 1741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:12:06.468268 (+ 951us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:12:06.470674 (+ 2406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:12:06.471100 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:12:06.471197 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:12:06.474960 (+ 3763us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:12:06.474964 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:12:06.474967 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:12:06.474999 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:12:06.477980 (+ 2981us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:12:06.478914 (+ 934us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:12:06.478919 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:12:06.478923 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:12:06.479101 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:12:06.479631 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:12:06.479636 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:12:06.479640 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:12:06.479996 (+ 356us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:12:06.480138 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0923 20:12:06.480943 (+ 805us) server_negotiation.cc:299] Negotiation successful
0923 20:12:06.481171 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":392,"thread_start_us":243,"threads_started":1}
I20260923 20:12:06.482461 14585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:12:06.461956 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.190:43227 (local address 127.1.170.131:58755)
0923 20:12:06.462765 (+ 809us) negotiation.cc:107] Waiting for socket to connect
0923 20:12:06.462821 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0923 20:12:06.465048 (+ 2227us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:12:06.465814 (+ 766us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:12:06.465822 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:12:06.466413 (+ 591us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:12:06.467087 (+ 674us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:12:06.467128 (+ 41us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:12:06.468454 (+ 1326us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:12:06.468460 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:12:06.470529 (+ 2069us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:12:06.470544 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:12:06.471317 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:12:06.471324 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:12:06.471614 (+ 290us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:12:06.472334 (+ 720us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:12:06.472363 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:12:06.474733 (+ 2370us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:12:06.478154 (+ 3421us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:12:06.478164 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:12:06.478181 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:12:06.478769 (+ 588us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:12:06.479231 (+ 462us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:12:06.479236 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:12:06.479240 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:12:06.479483 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:12:06.480128 (+ 645us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:12:06.480139 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:12:06.480560 (+ 421us) client_negotiation.cc:769] Sending connection context
0923 20:12:06.480813 (+ 253us) client_negotiation.cc:240] Negotiation successful
0923 20:12:06.481213 (+ 400us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":505,"thread_start_us":304,"threads_started":1}
I20260923 20:12:06.483940 14582 heartbeater.cc:347] Connected to a master server at 127.1.170.190:43227
I20260923 20:12:06.484246 14582 heartbeater.cc:464] Registering TS with master...
I20260923 20:12:06.485020 14582 heartbeater.cc:511] Master 127.1.170.190:43227 requested a full tablet report, sending...
I20260923 20:12:06.486580 14092 ts_manager.cc:194] Registered new tserver with Master: ec85741e5ec64d42a20e3c914e696a87 (127.1.170.131:34661)
I20260923 20:12:06.487421 14092 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.170.131@KRBTEST.COM'} at 127.1.170.131:58755
I20260923 20:12:06.493059 1706 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260923 20:12:06.495532 1706 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskqUuku6/build/asan/bin/kudu pid 14172
Sep 23 20:12:06 dist-test-slave-th4g krb5kdc[13616](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194322, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.170.129@KRBTEST.COM
I20260923 20:12:06.514618 14587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:12:06.496477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49612 (local address 127.1.170.129:44269)
0923 20:12:06.497402 (+ 925us) server_negotiation.cc:206] Beginning negotiation
0923 20:12:06.497410 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 20:12:06.497435 (+ 25us) server_negotiation.cc:407] Connection header received
0923 20:12:06.497491 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:12:06.497498 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:12:06.497723 (+ 225us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:12:06.497934 (+ 211us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:12:06.499495 (+ 1561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:12:06.500609 (+ 1114us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:12:06.502624 (+ 2015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:12:06.503267 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:12:06.503424 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:12:06.508336 (+ 4912us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:12:06.508349 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:12:06.508364 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:12:06.508402 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:12:06.511458 (+ 3056us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:12:06.512288 (+ 830us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:12:06.512295 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:12:06.512298 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:12:06.512435 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:12:06.512861 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:12:06.512870 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:12:06.512873 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:12:06.513249 (+ 376us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:12:06.513593 (+ 344us) server_negotiation.cc:1035] Waiting for connection context
0923 20:12:06.513938 (+ 345us) server_negotiation.cc:299] Negotiation successful
0923 20:12:06.514248 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":739,"thread_start_us":530,"threads_started":1}
I20260923 20:12:06.515928 14300 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:12:06.624641 1706 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqUuku6/build/asan/bin/kudu with pid 14172
I20260923 20:12:06.642168 1706 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskqUuku6/build/asan/bin/kudu pid 14312
Sep 23 20:12:06 dist-test-slave-th4g krb5kdc[13616](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194322, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.170.130@KRBTEST.COM
I20260923 20:12:06.662231 14589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:12:06.643117 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58044 (local address 127.1.170.130:45645)
0923 20:12:06.643693 (+ 576us) server_negotiation.cc:206] Beginning negotiation
0923 20:12:06.643701 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 20:12:06.643726 (+ 25us) server_negotiation.cc:407] Connection header received
0923 20:12:06.643787 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:12:06.643794 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:12:06.643998 (+ 204us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:12:06.644126 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:12:06.645358 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:12:06.646397 (+ 1039us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:12:06.647611 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:12:06.648124 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:12:06.648229 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:12:06.656588 (+ 8359us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:12:06.656601 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:12:06.656616 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:12:06.656672 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:12:06.659592 (+ 2920us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:12:06.660398 (+ 806us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:12:06.660405 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:12:06.660408 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:12:06.660535 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:12:06.660856 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:12:06.660859 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:12:06.660861 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:12:06.661179 (+ 318us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:12:06.661402 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0923 20:12:06.661602 (+ 200us) server_negotiation.cc:299] Negotiation successful
0923 20:12:06.661811 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":424,"thread_start_us":287,"threads_started":1}
I20260923 20:12:06.663445 14443 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:12:06.744915 1706 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqUuku6/build/asan/bin/kudu with pid 14312
I20260923 20:12:06.762215 1706 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskqUuku6/build/asan/bin/kudu pid 14454
Sep 23 20:12:06 dist-test-slave-th4g krb5kdc[13616](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194322, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.170.131@KRBTEST.COM
I20260923 20:12:06.778009 14591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:12:06.763191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33196 (local address 127.1.170.131:34661)
0923 20:12:06.763887 (+ 696us) server_negotiation.cc:206] Beginning negotiation
0923 20:12:06.763894 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:12:06.763931 (+ 37us) server_negotiation.cc:407] Connection header received
0923 20:12:06.764027 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:12:06.764033 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:12:06.764251 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:12:06.764392 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:12:06.765563 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:12:06.766646 (+ 1083us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:12:06.767918 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:12:06.768455 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:12:06.768551 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:12:06.772754 (+ 4203us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:12:06.772764 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:12:06.772778 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:12:06.772811 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:12:06.775547 (+ 2736us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:12:06.776192 (+ 645us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:12:06.776199 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:12:06.776201 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:12:06.776319 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:12:06.776619 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:12:06.776621 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:12:06.776623 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:12:06.776915 (+ 292us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:12:06.777127 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0923 20:12:06.777377 (+ 250us) server_negotiation.cc:299] Negotiation successful
0923 20:12:06.777551 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":551,"thread_start_us":409,"threads_started":1}
I20260923 20:12:06.779338 14577 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:12:06.860817 1706 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqUuku6/build/asan/bin/kudu with pid 14454
I20260923 20:12:06.876286 1706 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskqUuku6/build/asan/bin/kudu pid 14060
I20260923 20:12:06.877458 14122 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:12:06.984314 1706 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqUuku6/build/asan/bin/kudu with pid 14060
I20260923 20:12:07.005757 1706 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqUuku6/build/asan/bin/kudu
/tmp/dist-test-taskqUuku6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194121577027-1706-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194121577027-1706-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194121577027-1706-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194121577027-1706-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.170.190:43227
--webserver_interface=127.1.170.190
--webserver_port=37477
--keytab_file=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194121577027-1706-0/krb5kdc/kudu_127.1.170.190.keytab
--principal=kudu/127.1.170.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.170.148:34035
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.170.190:43227
--ranger_config_path=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194121577027-1706-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.1.170.148:39063/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.1.170.148:39063/kms with env {KRB5CCNAME=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194121577027-1706-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194121577027-1706-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194121577027-1706-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:12:07.327960 14594 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:12:07.328227 14594 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:12:07.328372 14594 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:12:07.335747 14594 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 20:12:07.335853 14594 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:12:07.335937 14594 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:12:07.336015 14594 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 20:12:07.336082 14594 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 20:12:07.346158 14594 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.170.148:34035
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194121577027-1706-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194121577027-1706-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.1.170.148:39063/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.1.170.190:43227
--ranger_config_path=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194121577027-1706-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194121577027-1706-0/krb5kdc/kudu_127.1.170.190.keytab
--principal=kudu/127.1.170.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.170.190:43227
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194121577027-1706-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.170.190
--webserver_port=37477
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194121577027-1706-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:12:07.347352 14594 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:12:07.348608 14594 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:12:07.359818 14599 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:12:07.359845 14600 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:12:07.360085 14602 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:12:07.360617 14594 server_base.cc:1034] running on GCE node
I20260923 20:12:07.361269 14594 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:12:07.362608 14594 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:12:07.363821 14594 hybrid_clock.cc:648] HybridClock initialized: now 1790194327363782 us; error 33 us; skew 500 ppm
I20260923 20:12:07.364180 14594 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:12:07 dist-test-slave-th4g krb5kdc[13616](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194327, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.170.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:12:07.367384 14594 init.cc:377] Logged in from keytab as kudu/127.1.170.190@KRBTEST.COM (short username kudu)
I20260923 20:12:07.368755 14594 webserver.cc:460] Webserver started at http://127.1.170.190:37477/ using document root <none> and password file <none>
I20260923 20:12:07.369343 14594 fs_manager.cc:359] Metadata directory not provided
I20260923 20:12:07.369446 14594 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 23 20:12:07 dist-test-slave-th4g krb5kdc[13616](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194327, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.170.190@KRBTEST.COM for HTTP/127.1.170.148@KRBTEST.COM
I20260923 20:12:07.429478 14594 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.004s sys 0.000s
I20260923 20:12:07.433092 14610 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:12:07.434716 14594 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260923 20:12:07.434891 14594 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194121577027-1706-0/minicluster-data/master-0/data,/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194121577027-1706-0/minicluster-data/master-0/wal
uuid: "528212d773b24273adebf10a109a5abe"
format_stamp: "Formatted at 2026-09-23 20:12:03 on dist-test-slave-th4g"
server_key: "644f02c5d8228a4e0406124d68f025bb"
server_key_iv: "113a11d15ecbff116130b0f6319b21fc"
server_key_version: "kuduclusterkey@0"
I20260923 20:12:07.436174 14594 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194121577027-1706-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194121577027-1706-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194121577027-1706-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:12:07.457749 14594 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:12:07.461526 14594 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:12:07.461763 14594 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:12:07.481040 14594 rpc_server.cc:307] RPC server started. Bound to: 127.1.170.190:43227
I20260923 20:12:07.481063 14661 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.170.190:43227 every 8 connection(s)
I20260923 20:12:07.482616 14594 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194121577027-1706-0/minicluster-data/master-0/data/info.pb
I20260923 20:12:07.489177 1706 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqUuku6/build/asan/bin/kudu as pid 14594
I20260923 20:12:07.489619 14662 sys_catalog.cc:263] Verifying existing consensus state
I20260923 20:12:07.490267 1706 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqUuku6/build/asan/bin/kudu
/tmp/dist-test-taskqUuku6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194121577027-1706-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194121577027-1706-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194121577027-1706-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194121577027-1706-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.1.170.148:39063/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.170.129:44269
--local_ip_for_outbound_sockets=127.1.170.129
--tserver_master_addrs=127.1.170.190:43227
--webserver_port=39207
--webserver_interface=127.1.170.129
--keytab_file=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194121577027-1706-0/krb5kdc/kudu_127.1.170.129.keytab
--principal=kudu/127.1.170.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.170.148:34035
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.1.170.148:39063/kms with env {KRB5CCNAME=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194121577027-1706-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194121577027-1706-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194121577027-1706-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260923 20:12:07.494979 14662 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 528212d773b24273adebf10a109a5abe: Bootstrap starting.
I20260923 20:12:07.514971 14662 log.cc:824] T 00000000000000000000000000000000 P 528212d773b24273adebf10a109a5abe: Log is configured to *not* fsync() on all Append() calls
I20260923 20:12:07.524113 14662 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 528212d773b24273adebf10a109a5abe: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260923 20:12:07.524658 14662 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 528212d773b24273adebf10a109a5abe: Bootstrap complete.
I20260923 20:12:07.533807 14662 raft_consensus.cc:348] T 00000000000000000000000000000000 P 528212d773b24273adebf10a109a5abe [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "528212d773b24273adebf10a109a5abe" member_type: VOTER last_known_addr { host: "127.1.170.190" port: 43227 } }
I20260923 20:12:07.534730 14662 raft_consensus.cc:729] T 00000000000000000000000000000000 P 528212d773b24273adebf10a109a5abe [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 528212d773b24273adebf10a109a5abe, State: Initialized, Role: FOLLOWER
I20260923 20:12:07.535377 14662 consensus_queue.cc:260] T 00000000000000000000000000000000 P 528212d773b24273adebf10a109a5abe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "528212d773b24273adebf10a109a5abe" member_type: VOTER last_known_addr { host: "127.1.170.190" port: 43227 } }
I20260923 20:12:07.535610 14662 raft_consensus.cc:388] T 00000000000000000000000000000000 P 528212d773b24273adebf10a109a5abe [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 20:12:07.535749 14662 raft_consensus.cc:482] T 00000000000000000000000000000000 P 528212d773b24273adebf10a109a5abe [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 20:12:07.535929 14662 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 528212d773b24273adebf10a109a5abe [term 1 FOLLOWER]: Advancing to term 2
I20260923 20:12:07.540009 14662 raft_consensus.cc:504] T 00000000000000000000000000000000 P 528212d773b24273adebf10a109a5abe [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "528212d773b24273adebf10a109a5abe" member_type: VOTER last_known_addr { host: "127.1.170.190" port: 43227 } }
I20260923 20:12:07.540586 14662 leader_election.cc:302] T 00000000000000000000000000000000 P 528212d773b24273adebf10a109a5abe [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 528212d773b24273adebf10a109a5abe; no voters:
I20260923 20:12:07.541368 14662 leader_election.cc:288] T 00000000000000000000000000000000 P 528212d773b24273adebf10a109a5abe [CANDIDATE]: Term 2 election: Requested vote from peers
I20260923 20:12:07.541535 14666 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 528212d773b24273adebf10a109a5abe [term 2 FOLLOWER]: Leader election won for term 2
I20260923 20:12:07.543989 14662 sys_catalog.cc:564] T 00000000000000000000000000000000 P 528212d773b24273adebf10a109a5abe [sys.catalog]: configured and running, proceeding with master startup.
I20260923 20:12:07.547772 14666 raft_consensus.cc:686] T 00000000000000000000000000000000 P 528212d773b24273adebf10a109a5abe [term 2 LEADER]: Becoming Leader. State: Replica: 528212d773b24273adebf10a109a5abe, State: Running, Role: LEADER
I20260923 20:12:07.548484 14666 consensus_queue.cc:237] T 00000000000000000000000000000000 P 528212d773b24273adebf10a109a5abe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "528212d773b24273adebf10a109a5abe" member_type: VOTER last_known_addr { host: "127.1.170.190" port: 43227 } }
I20260923 20:12:07.551434 14667 sys_catalog.cc:455] T 00000000000000000000000000000000 P 528212d773b24273adebf10a109a5abe [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "528212d773b24273adebf10a109a5abe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "528212d773b24273adebf10a109a5abe" member_type: VOTER last_known_addr { host: "127.1.170.190" port: 43227 } } }
I20260923 20:12:07.551503 14668 sys_catalog.cc:455] T 00000000000000000000000000000000 P 528212d773b24273adebf10a109a5abe [sys.catalog]: SysCatalogTable state changed. Reason: New leader 528212d773b24273adebf10a109a5abe. Latest consensus state: current_term: 2 leader_uuid: "528212d773b24273adebf10a109a5abe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "528212d773b24273adebf10a109a5abe" member_type: VOTER last_known_addr { host: "127.1.170.190" port: 43227 } } }
I20260923 20:12:07.551847 14667 sys_catalog.cc:458] T 00000000000000000000000000000000 P 528212d773b24273adebf10a109a5abe [sys.catalog]: This master's current role is: LEADER
I20260923 20:12:07.551860 14668 sys_catalog.cc:458] T 00000000000000000000000000000000 P 528212d773b24273adebf10a109a5abe [sys.catalog]: This master's current role is: LEADER
I20260923 20:12:07.553334 14662 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194121577027-1706-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260923 20:12:07.562685 14681 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 20:12:07.575740 14681 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 20:12:07.577672 14681 catalog_manager.cc:1259] Loaded cluster ID: 59f84a8184324d128eed7530192a26c4
I20260923 20:12:07.577908 14681 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 20:12:07.583035 14681 catalog_manager.cc:1504] Loading token signing keys...
I20260923 20:12:07.584944 14681 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 528212d773b24273adebf10a109a5abe: Loaded TSK: 0
I20260923 20:12:07.586760 14681 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260923 20:12:07.936226 14664 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:12:07.936708 14664 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:12:07.936911 14664 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:12:07.948607 14664 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:12:07.948843 14664 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:12:07.949107 14664 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.170.129
I20260923 20:12:07.964895 14664 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.170.148:34035
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194121577027-1706-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194121577027-1706-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.1.170.148:39063/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194121577027-1706-0/krb5kdc/kudu_127.1.170.129.keytab
--principal=kudu/127.1.170.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.170.129:44269
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194121577027-1706-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.170.129
--webserver_port=39207
--tserver_master_addrs=127.1.170.190:43227
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.170.129
--log_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194121577027-1706-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:12:07.966832 14664 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:12:07.968524 14664 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:12:08.009644 14699 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:12:08.010267 14700 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:12:08.017023 14702 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:12:08.017947 14664 server_base.cc:1034] running on GCE node
I20260923 20:12:08.018931 14664 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:12:08.022714 14664 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:12:08.024012 14664 hybrid_clock.cc:648] HybridClock initialized: now 1790194328024006 us; error 141 us; skew 500 ppm
I20260923 20:12:08.024356 14664 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:12:08 dist-test-slave-th4g krb5kdc[13616](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194328, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.170.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:12:08.027886 14664 init.cc:377] Logged in from keytab as kudu/127.1.170.129@KRBTEST.COM (short username kudu)
I20260923 20:12:08.031823 14664 webserver.cc:460] Webserver started at http://127.1.170.129:39207/ using document root <none> and password file <none>
I20260923 20:12:08.032573 14664 fs_manager.cc:359] Metadata directory not provided
I20260923 20:12:08.032740 14664 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 23 20:12:08 dist-test-slave-th4g krb5kdc[13616](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194328, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.170.129@KRBTEST.COM for HTTP/127.1.170.148@KRBTEST.COM
I20260923 20:12:08.076943 14664 fs_manager.cc:709] Time spent opening directory manager: real 0.015s user 0.006s sys 0.001s
I20260923 20:12:08.091795 14710 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:12:08.094035 14664 fs_manager.cc:725] Time spent opening block manager: real 0.015s user 0.003s sys 0.011s
I20260923 20:12:08.094254 14664 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194121577027-1706-0/minicluster-data/ts-0/data,/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194121577027-1706-0/minicluster-data/ts-0/wal
uuid: "711ac19c204c49e5842f923bebce66cc"
format_stamp: "Formatted at 2026-09-23 20:12:04 on dist-test-slave-th4g"
server_key: "f34add8cd89b65ca991f0eccd3468edb"
server_key_iv: "ed6bddd757467a2bdce1f7300dd41ac4"
server_key_version: "kuduclusterkey@0"
I20260923 20:12:08.095728 14664 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194121577027-1706-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194121577027-1706-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194121577027-1706-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:12:08.127770 14664 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:12:08.133368 14664 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:12:08.133764 14664 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:12:08.134994 14664 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 20:12:08.136797 14664 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 20:12:08.136902 14664 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 20:12:08.137006 14664 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 20:12:08.137079 14664 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 20:12:08.206351 14664 rpc_server.cc:307] RPC server started. Bound to: 127.1.170.129:44269
I20260923 20:12:08.206409 14823 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.170.129:44269 every 8 connection(s)
I20260923 20:12:08.208179 14664 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194121577027-1706-0/minicluster-data/ts-0/data/info.pb
I20260923 20:12:08.217480 1706 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqUuku6/build/asan/bin/kudu as pid 14664
I20260923 20:12:08.218354 1706 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqUuku6/build/asan/bin/kudu
/tmp/dist-test-taskqUuku6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194121577027-1706-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194121577027-1706-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194121577027-1706-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194121577027-1706-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.1.170.148:39063/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.170.130:45645
--local_ip_for_outbound_sockets=127.1.170.130
--tserver_master_addrs=127.1.170.190:43227
--webserver_port=36041
--webserver_interface=127.1.170.130
--keytab_file=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194121577027-1706-0/krb5kdc/kudu_127.1.170.130.keytab
--principal=kudu/127.1.170.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.170.148:34035
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.1.170.148:39063/kms with env {KRB5CCNAME=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194121577027-1706-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194121577027-1706-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194121577027-1706-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 23 20:12:08 dist-test-slave-th4g krb5kdc[13616](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194328, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.170.129@KRBTEST.COM for kudu/127.1.170.190@KRBTEST.COM
I20260923 20:12:08.242063 14826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:12:08.211695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.170.129:38851 (local address 127.1.170.190:43227)
0923 20:12:08.212475 (+ 780us) server_negotiation.cc:206] Beginning negotiation
0923 20:12:08.212490 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0923 20:12:08.213160 (+ 670us) server_negotiation.cc:407] Connection header received
0923 20:12:08.216013 (+ 2853us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:12:08.216029 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:12:08.216385 (+ 356us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:12:08.216768 (+ 383us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:12:08.218464 (+ 1696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:12:08.220256 (+ 1792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:12:08.222412 (+ 2156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:12:08.223543 (+ 1131us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:12:08.223734 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:12:08.229037 (+ 5303us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:12:08.229050 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:12:08.229073 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:12:08.229133 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:12:08.235399 (+ 6266us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:12:08.236555 (+ 1156us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:12:08.236568 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:12:08.236578 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:12:08.236787 (+ 209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:12:08.237380 (+ 593us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:12:08.237387 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:12:08.237393 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:12:08.238169 (+ 776us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:12:08.238366 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0923 20:12:08.239871 (+ 1505us) server_negotiation.cc:299] Negotiation successful
0923 20:12:08.240340 (+ 469us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":553,"thread_start_us":333,"threads_started":1}
I20260923 20:12:08.244386 14827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:12:08.211800 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.190:43227 (local address 127.1.170.129:38851)
0923 20:12:08.212967 (+ 1167us) negotiation.cc:107] Waiting for socket to connect
0923 20:12:08.213005 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0923 20:12:08.215348 (+ 2343us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:12:08.216946 (+ 1598us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:12:08.216960 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:12:08.217533 (+ 573us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:12:08.218241 (+ 708us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:12:08.218261 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:12:08.220452 (+ 2191us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:12:08.220464 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:12:08.222211 (+ 1747us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:12:08.222230 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:12:08.224236 (+ 2006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:12:08.224246 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:12:08.224552 (+ 306us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:12:08.225476 (+ 924us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:12:08.225510 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:12:08.228615 (+ 3105us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:12:08.235609 (+ 6994us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:12:08.235623 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:12:08.235644 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:12:08.236341 (+ 697us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:12:08.236944 (+ 603us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:12:08.236951 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:12:08.236958 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:12:08.237234 (+ 276us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:12:08.238442 (+ 1208us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:12:08.238454 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:12:08.238928 (+ 474us) client_negotiation.cc:769] Sending connection context
0923 20:12:08.239267 (+ 339us) client_negotiation.cc:240] Negotiation successful
0923 20:12:08.242960 (+ 3693us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":923,"thread_start_us":720,"threads_started":1}
I20260923 20:12:08.248420 14824 heartbeater.cc:347] Connected to a master server at 127.1.170.190:43227
I20260923 20:12:08.248777 14824 heartbeater.cc:464] Registering TS with master...
I20260923 20:12:08.249764 14824 heartbeater.cc:511] Master 127.1.170.190:43227 requested a full tablet report, sending...
W20260923 20:12:08.253695 14824 heartbeater.cc:644] Failed to heartbeat to 127.1.170.190:43227 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260923 20:12:08.253993 14824 heartbeater.cc:464] Registering TS with master...
I20260923 20:12:08.254317 14824 heartbeater.cc:511] Master 127.1.170.190:43227 requested a full tablet report, sending...
I20260923 20:12:08.255723 14824 heartbeater.cc:464] Registering TS with master...
I20260923 20:12:08.256089 14824 heartbeater.cc:511] Master 127.1.170.190:43227 requested a full tablet report, sending...
W20260923 20:12:08.257256 14824 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260923 20:12:08.628214 14828 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:12:08.628625 14828 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:12:08.628783 14828 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:12:08.637974 14828 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:12:08.638092 14828 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:12:08.638283 14828 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.170.130
I20260923 20:12:08.650115 14828 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.170.148:34035
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194121577027-1706-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194121577027-1706-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.1.170.148:39063/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194121577027-1706-0/krb5kdc/kudu_127.1.170.130.keytab
--principal=kudu/127.1.170.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.170.130:45645
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194121577027-1706-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.170.130
--webserver_port=36041
--tserver_master_addrs=127.1.170.190:43227
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.170.130
--log_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194121577027-1706-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:12:08.651549 14828 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:12:08.652835 14828 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:12:08.667410 14843 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:12:08.667694 14842 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:12:08.668056 14845 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:12:08.668972 14828 server_base.cc:1034] running on GCE node
I20260923 20:12:08.669641 14828 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:12:08.670945 14828 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:12:08.672212 14828 hybrid_clock.cc:648] HybridClock initialized: now 1790194328672151 us; error 45 us; skew 500 ppm
I20260923 20:12:08.672583 14828 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:12:08 dist-test-slave-th4g krb5kdc[13616](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194328, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.170.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:12:08.677037 14828 init.cc:377] Logged in from keytab as kudu/127.1.170.130@KRBTEST.COM (short username kudu)
I20260923 20:12:08.679098 14828 webserver.cc:460] Webserver started at http://127.1.170.130:36041/ using document root <none> and password file <none>
I20260923 20:12:08.680016 14828 fs_manager.cc:359] Metadata directory not provided
I20260923 20:12:08.680172 14828 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 23 20:12:08 dist-test-slave-th4g krb5kdc[13616](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194328, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.170.130@KRBTEST.COM for HTTP/127.1.170.148@KRBTEST.COM
I20260923 20:12:08.718478 14828 fs_manager.cc:709] Time spent opening directory manager: real 0.021s user 0.007s sys 0.011s
I20260923 20:12:08.735088 14853 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:12:08.737145 14828 fs_manager.cc:725] Time spent opening block manager: real 0.017s user 0.004s sys 0.012s
I20260923 20:12:08.737355 14828 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194121577027-1706-0/minicluster-data/ts-1/data,/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194121577027-1706-0/minicluster-data/ts-1/wal
uuid: "0bcb955ca73c4fdd8b6b2307210c43ac"
format_stamp: "Formatted at 2026-09-23 20:12:05 on dist-test-slave-th4g"
server_key: "5541006d2c0f4667379ea0a331f17d64"
server_key_iv: "f224528856ed3e5b0b05580bcd1ed75a"
server_key_version: "kuduclusterkey@0"
I20260923 20:12:08.738811 14828 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194121577027-1706-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194121577027-1706-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194121577027-1706-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:12:08.764868 14828 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:12:08.770231 14828 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:12:08.770539 14828 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:12:08.771763 14828 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 20:12:08.773490 14828 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 20:12:08.773600 14828 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 20:12:08.773702 14828 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 20:12:08.773777 14828 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 20:12:08.811486 14828 rpc_server.cc:307] RPC server started. Bound to: 127.1.170.130:45645
I20260923 20:12:08.811661 14967 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.170.130:45645 every 8 connection(s)
I20260923 20:12:08.813361 14828 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194121577027-1706-0/minicluster-data/ts-1/data/info.pb
I20260923 20:12:08.821033 1706 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqUuku6/build/asan/bin/kudu as pid 14828
I20260923 20:12:08.821833 1706 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqUuku6/build/asan/bin/kudu
/tmp/dist-test-taskqUuku6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194121577027-1706-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194121577027-1706-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194121577027-1706-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194121577027-1706-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.1.170.148:39063/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.170.131:34661
--local_ip_for_outbound_sockets=127.1.170.131
--tserver_master_addrs=127.1.170.190:43227
--webserver_port=46865
--webserver_interface=127.1.170.131
--keytab_file=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194121577027-1706-0/krb5kdc/kudu_127.1.170.131.keytab
--principal=kudu/127.1.170.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.170.148:34035
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.1.170.148:39063/kms with env {KRB5CCNAME=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194121577027-1706-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194121577027-1706-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194121577027-1706-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 23 20:12:08 dist-test-slave-th4g krb5kdc[13616](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790194328, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.170.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 23 20:12:08 dist-test-slave-th4g krb5kdc[13616](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194328, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.170.130@KRBTEST.COM for kudu/127.1.170.190@KRBTEST.COM
I20260923 20:12:08.871587 14970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:12:08.816899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.170.130:44223 (local address 127.1.170.190:43227)
0923 20:12:08.820515 (+ 3616us) server_negotiation.cc:206] Beginning negotiation
0923 20:12:08.820521 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:12:08.847668 (+ 27147us) server_negotiation.cc:407] Connection header received
0923 20:12:08.851252 (+ 3584us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:12:08.851260 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:12:08.851368 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:12:08.851454 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:12:08.853844 (+ 2390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:12:08.855097 (+ 1253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:12:08.857570 (+ 2473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:12:08.858374 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:12:08.858522 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:12:08.863374 (+ 4852us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:12:08.863381 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:12:08.863386 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:12:08.863430 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:12:08.867375 (+ 3945us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:12:08.868555 (+ 1180us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:12:08.868562 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:12:08.868565 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:12:08.868720 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:12:08.869284 (+ 564us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:12:08.869290 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:12:08.869293 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:12:08.869686 (+ 393us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:12:08.869822 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0923 20:12:08.870919 (+ 1097us) server_negotiation.cc:299] Negotiation successful
0923 20:12:08.871199 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3431,"thread_start_us":3278,"threads_started":1}
I20260923 20:12:08.874289 14971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:12:08.817751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.190:43227 (local address 127.1.170.130:44223)
0923 20:12:08.836315 (+ 18564us) negotiation.cc:107] Waiting for socket to connect
0923 20:12:08.847493 (+ 11178us) client_negotiation.cc:174] Beginning negotiation
0923 20:12:08.850799 (+ 3306us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:12:08.852090 (+ 1291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:12:08.852103 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:12:08.852731 (+ 628us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:12:08.853592 (+ 861us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:12:08.853616 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:12:08.855320 (+ 1704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:12:08.855326 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:12:08.857396 (+ 2070us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:12:08.857412 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:12:08.858869 (+ 1457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:12:08.858877 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:12:08.859208 (+ 331us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:12:08.860104 (+ 896us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:12:08.860166 (+ 62us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:12:08.863072 (+ 2906us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:12:08.867617 (+ 4545us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:12:08.867628 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:12:08.867646 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:12:08.868394 (+ 748us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:12:08.868870 (+ 476us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:12:08.868875 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:12:08.868879 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:12:08.869151 (+ 272us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:12:08.869986 (+ 835us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:12:08.869994 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:12:08.870424 (+ 430us) client_negotiation.cc:769] Sending connection context
0923 20:12:08.872286 (+ 1862us) client_negotiation.cc:240] Negotiation successful
0923 20:12:08.872902 (+ 616us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18124,"thread_start_us":20055,"threads_started":1}
I20260923 20:12:08.876263 14968 heartbeater.cc:347] Connected to a master server at 127.1.170.190:43227
I20260923 20:12:08.876628 14968 heartbeater.cc:464] Registering TS with master...
I20260923 20:12:08.877517 14968 heartbeater.cc:511] Master 127.1.170.190:43227 requested a full tablet report, sending...
I20260923 20:12:08.879724 14627 ts_manager.cc:194] Registered new tserver with Master: 0bcb955ca73c4fdd8b6b2307210c43ac (127.1.170.130:45645)
I20260923 20:12:08.881588 14627 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.170.130@KRBTEST.COM'} at 127.1.170.130:44223
I20260923 20:12:09.257572 14824 heartbeater.cc:464] Registering TS with master...
I20260923 20:12:09.258095 14824 heartbeater.cc:511] Master 127.1.170.190:43227 requested a full tablet report, sending...
I20260923 20:12:09.259447 14627 ts_manager.cc:194] Registered new tserver with Master: 711ac19c204c49e5842f923bebce66cc (127.1.170.129:44269)
I20260923 20:12:09.260620 14627 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.170.129@KRBTEST.COM'} at 127.1.170.129:38851
W20260923 20:12:09.305297 14972 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:12:09.305733 14972 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:12:09.305923 14972 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:12:09.318375 14972 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:12:09.318544 14972 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:12:09.318746 14972 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.170.131
I20260923 20:12:09.338253 14972 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.170.148:34035
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194121577027-1706-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194121577027-1706-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.1.170.148:39063/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194121577027-1706-0/krb5kdc/kudu_127.1.170.131.keytab
--principal=kudu/127.1.170.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.170.131:34661
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194121577027-1706-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.170.131
--webserver_port=46865
--tserver_master_addrs=127.1.170.190:43227
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.170.131
--log_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194121577027-1706-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:12:09.340075 14972 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:12:09.341588 14972 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:12:09.358155 14978 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
Sep 23 20:12:09 dist-test-slave-th4g krb5kdc[13616](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194328, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.170.190@KRBTEST.COM for HTTP/127.1.170.148@KRBTEST.COM
W20260923 20:12:09.366048 14979 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:12:09.366071 14972 server_base.cc:1034] running on GCE node
W20260923 20:12:09.366917 14981 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:12:09.367764 14972 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:12:09.374677 14972 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:12:09.376377 14972 hybrid_clock.cc:648] HybridClock initialized: now 1790194329376287 us; error 65 us; skew 500 ppm
I20260923 20:12:09.376854 14972 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:12:09 dist-test-slave-th4g krb5kdc[13616](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194329, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.170.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:12:09.381475 14972 init.cc:377] Logged in from keytab as kudu/127.1.170.131@KRBTEST.COM (short username kudu)
I20260923 20:12:09.432498 14972 webserver.cc:460] Webserver started at http://127.1.170.131:46865/ using document root <none> and password file <none>
I20260923 20:12:09.433765 14972 fs_manager.cc:359] Metadata directory not provided
I20260923 20:12:09.434033 14972 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 23 20:12:09 dist-test-slave-th4g krb5kdc[13616](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194329, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.170.131@KRBTEST.COM for HTTP/127.1.170.148@KRBTEST.COM
I20260923 20:12:09.471700 14972 fs_manager.cc:709] Time spent opening directory manager: real 0.013s user 0.010s sys 0.001s
I20260923 20:12:09.474983 14989 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:12:09.476577 14972 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260923 20:12:09.476802 14972 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194121577027-1706-0/minicluster-data/ts-2/data,/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194121577027-1706-0/minicluster-data/ts-2/wal
uuid: "ec85741e5ec64d42a20e3c914e696a87"
format_stamp: "Formatted at 2026-09-23 20:12:06 on dist-test-slave-th4g"
server_key: "041b42b62f06df43bbd36f9495fa06a0"
server_key_iv: "970cf46e1a7f15a4e417525b5e649400"
server_key_version: "kuduclusterkey@0"
I20260923 20:12:09.477885 14972 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194121577027-1706-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194121577027-1706-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194121577027-1706-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:12:09.493500 14972 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:12:09.499469 14972 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:12:09.499904 14972 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:12:09.502517 14972 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 20:12:09.506820 14972 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 20:12:09.506953 14972 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 20:12:09.507074 14972 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 20:12:09.507148 14972 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 20:12:09.554170 14972 rpc_server.cc:307] RPC server started. Bound to: 127.1.170.131:34661
I20260923 20:12:09.554214 15101 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.170.131:34661 every 8 connection(s)
I20260923 20:12:09.555627 14972 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194121577027-1706-0/minicluster-data/ts-2/data/info.pb
I20260923 20:12:09.562709 1706 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqUuku6/build/asan/bin/kudu as pid 14972
Sep 23 20:12:09 dist-test-slave-th4g krb5kdc[13616](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194329, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.170.131@KRBTEST.COM for kudu/127.1.170.190@KRBTEST.COM
I20260923 20:12:09.583425 15107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:12:09.564275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50662 (local address 127.1.170.190:43227)
0923 20:12:09.564982 (+ 707us) server_negotiation.cc:206] Beginning negotiation
0923 20:12:09.564991 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 20:12:09.565016 (+ 25us) server_negotiation.cc:407] Connection header received
0923 20:12:09.565076 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:12:09.565083 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:12:09.565190 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:12:09.567424 (+ 2234us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:12:09.567649 (+ 225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:12:09.568766 (+ 1117us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:12:09.570685 (+ 1919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:12:09.571311 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:12:09.574700 (+ 3389us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:12:09.574999 (+ 299us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:12:09.575005 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:12:09.575009 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:12:09.575055 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:12:09.579395 (+ 4340us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:12:09.580345 (+ 950us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:12:09.580351 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:12:09.580354 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:12:09.580526 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:12:09.580953 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:12:09.580958 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:12:09.580961 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:12:09.581314 (+ 353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:12:09.582706 (+ 1392us) server_negotiation.cc:1035] Waiting for connection context
0923 20:12:09.582833 (+ 127us) server_negotiation.cc:299] Negotiation successful
0923 20:12:09.583060 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":560,"thread_start_us":420,"threads_started":1}
I20260923 20:12:09.583832 15104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:12:09.558864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.170.131:58961 (local address 127.1.170.190:43227)
0923 20:12:09.559454 (+ 590us) server_negotiation.cc:206] Beginning negotiation
0923 20:12:09.559485 (+ 31us) server_negotiation.cc:399] Waiting for connection header
0923 20:12:09.560941 (+ 1456us) server_negotiation.cc:407] Connection header received
0923 20:12:09.563227 (+ 2286us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:12:09.563233 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:12:09.563336 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:12:09.563458 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:12:09.565528 (+ 2070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:12:09.566736 (+ 1208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:12:09.569323 (+ 2587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:12:09.569982 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:12:09.570079 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:12:09.574931 (+ 4852us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:12:09.574940 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:12:09.574945 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:12:09.575008 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:12:09.579238 (+ 4230us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:12:09.580381 (+ 1143us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:12:09.580387 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:12:09.580390 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:12:09.580538 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:12:09.581159 (+ 621us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:12:09.581164 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:12:09.581167 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:12:09.581579 (+ 412us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:12:09.581725 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0923 20:12:09.583070 (+ 1345us) server_negotiation.cc:299] Negotiation successful
0923 20:12:09.583296 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":408,"thread_start_us":295,"threads_started":1}
I20260923 20:12:09.587036 15105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:12:09.559583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.190:43227 (local address 127.1.170.131:58961)
0923 20:12:09.560456 (+ 873us) negotiation.cc:107] Waiting for socket to connect
0923 20:12:09.560512 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0923 20:12:09.562787 (+ 2275us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:12:09.563776 (+ 989us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:12:09.563788 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:12:09.564417 (+ 629us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:12:09.565290 (+ 873us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:12:09.565320 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:12:09.566983 (+ 1663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:12:09.566988 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:12:09.569141 (+ 2153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:12:09.569158 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:12:09.570592 (+ 1434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:12:09.570601 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:12:09.570933 (+ 332us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:12:09.571789 (+ 856us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:12:09.571820 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:12:09.574593 (+ 2773us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:12:09.579457 (+ 4864us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:12:09.579467 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:12:09.579488 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:12:09.580221 (+ 733us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:12:09.580719 (+ 498us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:12:09.580724 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:12:09.580728 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:12:09.581014 (+ 286us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:12:09.581847 (+ 833us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:12:09.581857 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:12:09.582417 (+ 560us) client_negotiation.cc:769] Sending connection context
0923 20:12:09.584098 (+ 1681us) client_negotiation.cc:240] Negotiation successful
0923 20:12:09.585206 (+ 1108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":582,"thread_start_us":361,"threads_started":1}
I20260923 20:12:09.588809 15102 heartbeater.cc:347] Connected to a master server at 127.1.170.190:43227
I20260923 20:12:09.589159 15102 heartbeater.cc:464] Registering TS with master...
I20260923 20:12:09.590121 15102 heartbeater.cc:511] Master 127.1.170.190:43227 requested a full tablet report, sending...
I20260923 20:12:09.592067 14627 ts_manager.cc:194] Registered new tserver with Master: ec85741e5ec64d42a20e3c914e696a87 (127.1.170.131:34661)
I20260923 20:12:09.593201 14627 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.170.131@KRBTEST.COM'} at 127.1.170.131:58961
I20260923 20:12:09.599206 1706 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260923 20:12:09.621398 15107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:12:09.602941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50676 (local address 127.1.170.190:43227)
0923 20:12:09.603150 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0923 20:12:09.603156 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:12:09.603865 (+ 709us) server_negotiation.cc:407] Connection header received
0923 20:12:09.604016 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:12:09.604021 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:12:09.604119 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:12:09.604234 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:12:09.606406 (+ 2172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:12:09.607434 (+ 1028us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:12:09.609530 (+ 2096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:12:09.610253 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:12:09.610375 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:12:09.614199 (+ 3824us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:12:09.614207 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:12:09.614212 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:12:09.614264 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:12:09.617991 (+ 3727us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:12:09.618878 (+ 887us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:12:09.618884 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:12:09.618888 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:12:09.619044 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:12:09.619547 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:12:09.619554 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:12:09.619560 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:12:09.619962 (+ 402us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:12:09.620182 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0923 20:12:09.620800 (+ 618us) server_negotiation.cc:299] Negotiation successful
0923 20:12:09.621052 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260923 20:12:09.628553 14627 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50676:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 20:12:09.640890 14627 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
Sep 23 20:12:09 dist-test-slave-th4g krb5kdc[13616](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194328, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.170.190@KRBTEST.COM for HTTP/127.1.170.148@KRBTEST.COM
I20260923 20:12:09.734570 15120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:12:09.681759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.129:44269 (local address 127.0.0.1:49628)
0923 20:12:09.708532 (+ 26773us) negotiation.cc:107] Waiting for socket to connect
0923 20:12:09.708588 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0923 20:12:09.708922 (+ 334us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:12:09.711982 (+ 3060us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:12:09.711991 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:12:09.712027 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:12:09.720251 (+ 8224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:12:09.720265 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:12:09.722854 (+ 2589us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:12:09.722859 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:12:09.726343 (+ 3484us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:12:09.726368 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:12:09.733029 (+ 6661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:12:09.733040 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:12:09.733598 (+ 558us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:12:09.733724 (+ 126us) client_negotiation.cc:769] Sending connection context
0923 20:12:09.733852 (+ 128us) client_negotiation.cc:240] Negotiation successful
0923 20:12:09.734118 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26462,"mutex_wait_us":884,"thread_start_us":27542,"threads_started":1}
I20260923 20:12:09.735770 15119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:12:09.682404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49628 (local address 127.1.170.129:44269)
0923 20:12:09.704772 (+ 22368us) server_negotiation.cc:206] Beginning negotiation
0923 20:12:09.704788 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0923 20:12:09.708724 (+ 3936us) server_negotiation.cc:407] Connection header received
0923 20:12:09.711157 (+ 2433us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:12:09.711168 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:12:09.711409 (+ 241us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:12:09.711602 (+ 193us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:12:09.720883 (+ 9281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:12:09.722619 (+ 1736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:12:09.728719 (+ 6100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:12:09.732527 (+ 3808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:12:09.734858 (+ 2331us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:12:09.735156 (+ 298us) server_negotiation.cc:1035] Waiting for connection context
0923 20:12:09.735251 (+ 95us) server_negotiation.cc:299] Negotiation successful
0923 20:12:09.735426 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22107,"thread_start_us":21891,"threads_started":1}
I20260923 20:12:09.738274 15118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:12:09.683317 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.131:34661 (local address 127.0.0.1:33212)
0923 20:12:09.709564 (+ 26247us) negotiation.cc:107] Waiting for socket to connect
0923 20:12:09.709588 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0923 20:12:09.709723 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:12:09.719341 (+ 9618us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:12:09.719347 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:12:09.719395 (+ 48us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:12:09.719742 (+ 347us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:12:09.719752 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:12:09.722266 (+ 2514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:12:09.722272 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:12:09.733572 (+ 11300us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:12:09.733590 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:12:09.737382 (+ 3792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:12:09.737389 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:12:09.737574 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:12:09.737606 (+ 32us) client_negotiation.cc:769] Sending connection context
0923 20:12:09.737691 (+ 85us) client_negotiation.cc:240] Negotiation successful
0923 20:12:09.737839 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26096,"mutex_wait_us":45,"thread_start_us":28435,"threads_started":1}
I20260923 20:12:09.739100 15121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:12:09.683691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33212 (local address 127.1.170.131:34661)
0923 20:12:09.707628 (+ 23937us) server_negotiation.cc:206] Beginning negotiation
0923 20:12:09.707638 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0923 20:12:09.709909 (+ 2271us) server_negotiation.cc:407] Connection header received
0923 20:12:09.710025 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:12:09.710034 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:12:09.710317 (+ 283us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:12:09.710510 (+ 193us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:12:09.720010 (+ 9500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:12:09.722069 (+ 2059us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:12:09.733824 (+ 11755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:12:09.735787 (+ 1963us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:12:09.735914 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:12:09.737148 (+ 1234us) server_negotiation.cc:1035] Waiting for connection context
0923 20:12:09.738579 (+ 1431us) server_negotiation.cc:299] Negotiation successful
0923 20:12:09.738761 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23733,"thread_start_us":23948,"threads_started":1}
I20260923 20:12:09.740761 15123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:12:09.712546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58054 (local address 127.1.170.130:45645)
0923 20:12:09.729441 (+ 16895us) server_negotiation.cc:206] Beginning negotiation
0923 20:12:09.729451 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0923 20:12:09.730424 (+ 973us) server_negotiation.cc:407] Connection header received
0923 20:12:09.731150 (+ 726us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:12:09.731159 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:12:09.731610 (+ 451us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:12:09.731832 (+ 222us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:12:09.732822 (+ 990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:12:09.734480 (+ 1658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:12:09.737385 (+ 2905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:12:09.738997 (+ 1612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:12:09.739117 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:12:09.739526 (+ 409us) server_negotiation.cc:1035] Waiting for connection context
0923 20:12:09.740154 (+ 628us) server_negotiation.cc:299] Negotiation successful
0923 20:12:09.740346 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16686,"thread_start_us":16906,"threads_started":1}
I20260923 20:12:09.740486 14758 tablet_service.cc:1467] Processing CreateTablet for tablet b90eb6f0659f42ea99ec303527350a9e (DEFAULT_TABLE table=test-table [id=d24ec6f2e9dc42bfa64852a8f46e46ee]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 20:12:09.741613 15122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:12:09.712515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.130:45645 (local address 127.0.0.1:58054)
0923 20:12:09.730289 (+ 17774us) negotiation.cc:107] Waiting for socket to connect
0923 20:12:09.730312 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0923 20:12:09.730573 (+ 261us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:12:09.732163 (+ 1590us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:12:09.732214 (+ 51us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:12:09.732248 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:12:09.732601 (+ 353us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:12:09.732615 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:12:09.734932 (+ 2317us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:12:09.734939 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:12:09.737161 (+ 2222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:12:09.737188 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:12:09.739753 (+ 2565us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:12:09.739760 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:12:09.739936 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:12:09.739968 (+ 32us) client_negotiation.cc:769] Sending connection context
0923 20:12:09.741086 (+ 1118us) client_negotiation.cc:240] Negotiation successful
0923 20:12:09.741255 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17625,"thread_start_us":17341,"threads_started":1}
I20260923 20:12:09.742378 14758 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b90eb6f0659f42ea99ec303527350a9e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:12:09.743788 15037 tablet_service.cc:1467] Processing CreateTablet for tablet b90eb6f0659f42ea99ec303527350a9e (DEFAULT_TABLE table=test-table [id=d24ec6f2e9dc42bfa64852a8f46e46ee]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 20:12:09.745402 15037 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b90eb6f0659f42ea99ec303527350a9e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:12:09.746356 14903 tablet_service.cc:1467] Processing CreateTablet for tablet b90eb6f0659f42ea99ec303527350a9e (DEFAULT_TABLE table=test-table [id=d24ec6f2e9dc42bfa64852a8f46e46ee]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 20:12:09.747967 14903 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b90eb6f0659f42ea99ec303527350a9e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:12:09.771114 15125 tablet_bootstrap.cc:492] T b90eb6f0659f42ea99ec303527350a9e P 0bcb955ca73c4fdd8b6b2307210c43ac: Bootstrap starting.
I20260923 20:12:09.773042 15124 tablet_bootstrap.cc:492] T b90eb6f0659f42ea99ec303527350a9e P ec85741e5ec64d42a20e3c914e696a87: Bootstrap starting.
I20260923 20:12:09.777715 15125 tablet_bootstrap.cc:654] T b90eb6f0659f42ea99ec303527350a9e P 0bcb955ca73c4fdd8b6b2307210c43ac: Neither blocks nor log segments found. Creating new log.
I20260923 20:12:09.778687 15126 tablet_bootstrap.cc:492] T b90eb6f0659f42ea99ec303527350a9e P 711ac19c204c49e5842f923bebce66cc: Bootstrap starting.
I20260923 20:12:09.782924 15126 tablet_bootstrap.cc:654] T b90eb6f0659f42ea99ec303527350a9e P 711ac19c204c49e5842f923bebce66cc: Neither blocks nor log segments found. Creating new log.
I20260923 20:12:09.783735 15124 tablet_bootstrap.cc:654] T b90eb6f0659f42ea99ec303527350a9e P ec85741e5ec64d42a20e3c914e696a87: Neither blocks nor log segments found. Creating new log.
I20260923 20:12:09.784768 15125 log.cc:824] T b90eb6f0659f42ea99ec303527350a9e P 0bcb955ca73c4fdd8b6b2307210c43ac: Log is configured to *not* fsync() on all Append() calls
I20260923 20:12:09.785089 15124 log.cc:824] T b90eb6f0659f42ea99ec303527350a9e P ec85741e5ec64d42a20e3c914e696a87: Log is configured to *not* fsync() on all Append() calls
I20260923 20:12:09.787020 15126 log.cc:824] T b90eb6f0659f42ea99ec303527350a9e P 711ac19c204c49e5842f923bebce66cc: Log is configured to *not* fsync() on all Append() calls
I20260923 20:12:09.797217 15124 tablet_bootstrap.cc:492] T b90eb6f0659f42ea99ec303527350a9e P ec85741e5ec64d42a20e3c914e696a87: No bootstrap required, opened a new log
I20260923 20:12:09.797652 15124 ts_tablet_manager.cc:1397] T b90eb6f0659f42ea99ec303527350a9e P ec85741e5ec64d42a20e3c914e696a87: Time spent bootstrapping tablet: real 0.025s user 0.007s sys 0.008s
I20260923 20:12:09.800608 15126 tablet_bootstrap.cc:492] T b90eb6f0659f42ea99ec303527350a9e P 711ac19c204c49e5842f923bebce66cc: No bootstrap required, opened a new log
I20260923 20:12:09.801035 15126 ts_tablet_manager.cc:1397] T b90eb6f0659f42ea99ec303527350a9e P 711ac19c204c49e5842f923bebce66cc: Time spent bootstrapping tablet: real 0.023s user 0.012s sys 0.000s
I20260923 20:12:09.808944 15125 tablet_bootstrap.cc:492] T b90eb6f0659f42ea99ec303527350a9e P 0bcb955ca73c4fdd8b6b2307210c43ac: No bootstrap required, opened a new log
I20260923 20:12:09.809384 15125 ts_tablet_manager.cc:1397] T b90eb6f0659f42ea99ec303527350a9e P 0bcb955ca73c4fdd8b6b2307210c43ac: Time spent bootstrapping tablet: real 0.039s user 0.013s sys 0.004s
I20260923 20:12:09.809077 15124 raft_consensus.cc:348] T b90eb6f0659f42ea99ec303527350a9e P ec85741e5ec64d42a20e3c914e696a87 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "711ac19c204c49e5842f923bebce66cc" member_type: VOTER last_known_addr { host: "127.1.170.129" port: 44269 } } peers { permanent_uuid: "ec85741e5ec64d42a20e3c914e696a87" member_type: VOTER last_known_addr { host: "127.1.170.131" port: 34661 } } peers { permanent_uuid: "0bcb955ca73c4fdd8b6b2307210c43ac" member_type: VOTER last_known_addr { host: "127.1.170.130" port: 45645 } }
I20260923 20:12:09.809543 15124 raft_consensus.cc:374] T b90eb6f0659f42ea99ec303527350a9e P ec85741e5ec64d42a20e3c914e696a87 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:12:09.809659 15124 raft_consensus.cc:729] T b90eb6f0659f42ea99ec303527350a9e P ec85741e5ec64d42a20e3c914e696a87 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec85741e5ec64d42a20e3c914e696a87, State: Initialized, Role: FOLLOWER
I20260923 20:12:09.810483 15124 consensus_queue.cc:260] T b90eb6f0659f42ea99ec303527350a9e P ec85741e5ec64d42a20e3c914e696a87 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "711ac19c204c49e5842f923bebce66cc" member_type: VOTER last_known_addr { host: "127.1.170.129" port: 44269 } } peers { permanent_uuid: "ec85741e5ec64d42a20e3c914e696a87" member_type: VOTER last_known_addr { host: "127.1.170.131" port: 34661 } } peers { permanent_uuid: "0bcb955ca73c4fdd8b6b2307210c43ac" member_type: VOTER last_known_addr { host: "127.1.170.130" port: 45645 } }
I20260923 20:12:09.812505 15102 heartbeater.cc:503] Master 127.1.170.190:43227 was elected leader, sending a full tablet report...
I20260923 20:12:09.813216 15124 ts_tablet_manager.cc:1428] T b90eb6f0659f42ea99ec303527350a9e P ec85741e5ec64d42a20e3c914e696a87: Time spent starting tablet: real 0.015s user 0.015s sys 0.000s
I20260923 20:12:09.819756 15126 raft_consensus.cc:348] T b90eb6f0659f42ea99ec303527350a9e P 711ac19c204c49e5842f923bebce66cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "711ac19c204c49e5842f923bebce66cc" member_type: VOTER last_known_addr { host: "127.1.170.129" port: 44269 } } peers { permanent_uuid: "ec85741e5ec64d42a20e3c914e696a87" member_type: VOTER last_known_addr { host: "127.1.170.131" port: 34661 } } peers { permanent_uuid: "0bcb955ca73c4fdd8b6b2307210c43ac" member_type: VOTER last_known_addr { host: "127.1.170.130" port: 45645 } }
I20260923 20:12:09.820196 15126 raft_consensus.cc:374] T b90eb6f0659f42ea99ec303527350a9e P 711ac19c204c49e5842f923bebce66cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:12:09.820333 15126 raft_consensus.cc:729] T b90eb6f0659f42ea99ec303527350a9e P 711ac19c204c49e5842f923bebce66cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 711ac19c204c49e5842f923bebce66cc, State: Initialized, Role: FOLLOWER
I20260923 20:12:09.820978 15126 consensus_queue.cc:260] T b90eb6f0659f42ea99ec303527350a9e P 711ac19c204c49e5842f923bebce66cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "711ac19c204c49e5842f923bebce66cc" member_type: VOTER last_known_addr { host: "127.1.170.129" port: 44269 } } peers { permanent_uuid: "ec85741e5ec64d42a20e3c914e696a87" member_type: VOTER last_known_addr { host: "127.1.170.131" port: 34661 } } peers { permanent_uuid: "0bcb955ca73c4fdd8b6b2307210c43ac" member_type: VOTER last_known_addr { host: "127.1.170.130" port: 45645 } }
I20260923 20:12:09.822183 14824 heartbeater.cc:503] Master 127.1.170.190:43227 was elected leader, sending a full tablet report...
I20260923 20:12:09.821842 15125 raft_consensus.cc:348] T b90eb6f0659f42ea99ec303527350a9e P 0bcb955ca73c4fdd8b6b2307210c43ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "711ac19c204c49e5842f923bebce66cc" member_type: VOTER last_known_addr { host: "127.1.170.129" port: 44269 } } peers { permanent_uuid: "ec85741e5ec64d42a20e3c914e696a87" member_type: VOTER last_known_addr { host: "127.1.170.131" port: 34661 } } peers { permanent_uuid: "0bcb955ca73c4fdd8b6b2307210c43ac" member_type: VOTER last_known_addr { host: "127.1.170.130" port: 45645 } }
I20260923 20:12:09.822419 15125 raft_consensus.cc:374] T b90eb6f0659f42ea99ec303527350a9e P 0bcb955ca73c4fdd8b6b2307210c43ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:12:09.822537 15125 raft_consensus.cc:729] T b90eb6f0659f42ea99ec303527350a9e P 0bcb955ca73c4fdd8b6b2307210c43ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0bcb955ca73c4fdd8b6b2307210c43ac, State: Initialized, Role: FOLLOWER
I20260923 20:12:09.822897 15126 ts_tablet_manager.cc:1428] T b90eb6f0659f42ea99ec303527350a9e P 711ac19c204c49e5842f923bebce66cc: Time spent starting tablet: real 0.022s user 0.009s sys 0.004s
I20260923 20:12:09.823161 15125 consensus_queue.cc:260] T b90eb6f0659f42ea99ec303527350a9e P 0bcb955ca73c4fdd8b6b2307210c43ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "711ac19c204c49e5842f923bebce66cc" member_type: VOTER last_known_addr { host: "127.1.170.129" port: 44269 } } peers { permanent_uuid: "ec85741e5ec64d42a20e3c914e696a87" member_type: VOTER last_known_addr { host: "127.1.170.131" port: 34661 } } peers { permanent_uuid: "0bcb955ca73c4fdd8b6b2307210c43ac" member_type: VOTER last_known_addr { host: "127.1.170.130" port: 45645 } }
I20260923 20:12:09.828487 14968 heartbeater.cc:503] Master 127.1.170.190:43227 was elected leader, sending a full tablet report...
I20260923 20:12:09.829249 15125 ts_tablet_manager.cc:1428] T b90eb6f0659f42ea99ec303527350a9e P 0bcb955ca73c4fdd8b6b2307210c43ac: Time spent starting tablet: real 0.020s user 0.006s sys 0.011s
W20260923 20:12:09.961862 14825 tablet.cc:2369] T b90eb6f0659f42ea99ec303527350a9e P 711ac19c204c49e5842f923bebce66cc: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260923 20:12:10.059660 15103 tablet.cc:2369] T b90eb6f0659f42ea99ec303527350a9e P ec85741e5ec64d42a20e3c914e696a87: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260923 20:12:10.073698 14969 tablet.cc:2369] T b90eb6f0659f42ea99ec303527350a9e P 0bcb955ca73c4fdd8b6b2307210c43ac: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 20:12:10.156718 15130 raft_consensus.cc:482] T b90eb6f0659f42ea99ec303527350a9e P ec85741e5ec64d42a20e3c914e696a87 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 20:12:10.157073 15130 raft_consensus.cc:504] T b90eb6f0659f42ea99ec303527350a9e P ec85741e5ec64d42a20e3c914e696a87 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "711ac19c204c49e5842f923bebce66cc" member_type: VOTER last_known_addr { host: "127.1.170.129" port: 44269 } } peers { permanent_uuid: "ec85741e5ec64d42a20e3c914e696a87" member_type: VOTER last_known_addr { host: "127.1.170.131" port: 34661 } } peers { permanent_uuid: "0bcb955ca73c4fdd8b6b2307210c43ac" member_type: VOTER last_known_addr { host: "127.1.170.130" port: 45645 } }
I20260923 20:12:10.158706 15130 leader_election.cc:288] T b90eb6f0659f42ea99ec303527350a9e P ec85741e5ec64d42a20e3c914e696a87 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 711ac19c204c49e5842f923bebce66cc (127.1.170.129:44269), 0bcb955ca73c4fdd8b6b2307210c43ac (127.1.170.130:45645)
I20260923 20:12:10.168483 15119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:12:10.159266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.170.131:54397 (local address 127.1.170.129:44269)
0923 20:12:10.159507 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0923 20:12:10.159516 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 20:12:10.160077 (+ 561us) server_negotiation.cc:407] Connection header received
0923 20:12:10.160266 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:12:10.160276 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:12:10.160366 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:12:10.160478 (+ 112us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:12:10.161259 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:12:10.162399 (+ 1140us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:12:10.165746 (+ 3347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:12:10.167059 (+ 1313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:12:10.167254 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:12:10.167332 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0923 20:12:10.168041 (+ 709us) server_negotiation.cc:299] Negotiation successful
0923 20:12:10.168236 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260923 20:12:10.168480 15123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:12:10.159370 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.170.131:60327 (local address 127.1.170.130:45645)
0923 20:12:10.159600 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0923 20:12:10.159606 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:12:10.160015 (+ 409us) server_negotiation.cc:407] Connection header received
0923 20:12:10.160271 (+ 256us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:12:10.160275 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:12:10.160335 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:12:10.160385 (+ 50us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:12:10.161259 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:12:10.162399 (+ 1140us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:12:10.165746 (+ 3347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:12:10.167099 (+ 1353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:12:10.167254 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:12:10.167331 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0923 20:12:10.168041 (+ 710us) server_negotiation.cc:299] Negotiation successful
0923 20:12:10.168187 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260923 20:12:10.168885 15141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:12:10.159073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.130:45645 (local address 127.1.170.131:60327)
0923 20:12:10.159877 (+ 804us) negotiation.cc:107] Waiting for socket to connect
0923 20:12:10.159899 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0923 20:12:10.160073 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:12:10.160591 (+ 518us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:12:10.160595 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:12:10.160633 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:12:10.161051 (+ 418us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:12:10.161061 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:12:10.162593 (+ 1532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:12:10.162599 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:12:10.165502 (+ 2903us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:12:10.165525 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:12:10.167472 (+ 1947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:12:10.167486 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:12:10.167789 (+ 303us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:12:10.167867 (+ 78us) client_negotiation.cc:769] Sending connection context
0923 20:12:10.167970 (+ 103us) client_negotiation.cc:240] Negotiation successful
0923 20:12:10.168220 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":659,"mutex_wait_us":304,"thread_start_us":582,"threads_started":1}
I20260923 20:12:10.169272 15140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:12:10.159073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.129:44269 (local address 127.1.170.131:54397)
0923 20:12:10.159964 (+ 891us) negotiation.cc:107] Waiting for socket to connect
0923 20:12:10.159981 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0923 20:12:10.160121 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:12:10.160527 (+ 406us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:12:10.160535 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:12:10.160567 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:12:10.161024 (+ 457us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:12:10.161042 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:12:10.162593 (+ 1551us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:12:10.162599 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:12:10.165502 (+ 2903us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:12:10.165525 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:12:10.167372 (+ 1847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:12:10.167382 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:12:10.167780 (+ 398us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:12:10.167858 (+ 78us) client_negotiation.cc:769] Sending connection context
0923 20:12:10.167970 (+ 112us) client_negotiation.cc:240] Negotiation successful
0923 20:12:10.168220 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":787,"mutex_wait_us":459,"spinlock_wait_cycles":23936,"thread_start_us":303,"threads_started":1}
I20260923 20:12:10.170122 14923 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b90eb6f0659f42ea99ec303527350a9e" candidate_uuid: "ec85741e5ec64d42a20e3c914e696a87" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0bcb955ca73c4fdd8b6b2307210c43ac" is_pre_election: true
I20260923 20:12:10.170234 14779 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b90eb6f0659f42ea99ec303527350a9e" candidate_uuid: "ec85741e5ec64d42a20e3c914e696a87" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "711ac19c204c49e5842f923bebce66cc" is_pre_election: true
I20260923 20:12:10.170645 14923 raft_consensus.cc:2454] T b90eb6f0659f42ea99ec303527350a9e P 0bcb955ca73c4fdd8b6b2307210c43ac [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ec85741e5ec64d42a20e3c914e696a87 in term 0.
I20260923 20:12:10.170739 14779 raft_consensus.cc:2454] T b90eb6f0659f42ea99ec303527350a9e P 711ac19c204c49e5842f923bebce66cc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ec85741e5ec64d42a20e3c914e696a87 in term 0.
I20260923 20:12:10.171483 14990 leader_election.cc:302] T b90eb6f0659f42ea99ec303527350a9e P ec85741e5ec64d42a20e3c914e696a87 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 0bcb955ca73c4fdd8b6b2307210c43ac, 711ac19c204c49e5842f923bebce66cc, ec85741e5ec64d42a20e3c914e696a87; no voters:
I20260923 20:12:10.171940 15130 raft_consensus.cc:2781] T b90eb6f0659f42ea99ec303527350a9e P ec85741e5ec64d42a20e3c914e696a87 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 20:12:10.172078 15130 raft_consensus.cc:482] T b90eb6f0659f42ea99ec303527350a9e P ec85741e5ec64d42a20e3c914e696a87 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 20:12:10.172216 15130 raft_consensus.cc:3037] T b90eb6f0659f42ea99ec303527350a9e P ec85741e5ec64d42a20e3c914e696a87 [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:12:10.174669 15130 raft_consensus.cc:504] T b90eb6f0659f42ea99ec303527350a9e P ec85741e5ec64d42a20e3c914e696a87 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "711ac19c204c49e5842f923bebce66cc" member_type: VOTER last_known_addr { host: "127.1.170.129" port: 44269 } } peers { permanent_uuid: "ec85741e5ec64d42a20e3c914e696a87" member_type: VOTER last_known_addr { host: "127.1.170.131" port: 34661 } } peers { permanent_uuid: "0bcb955ca73c4fdd8b6b2307210c43ac" member_type: VOTER last_known_addr { host: "127.1.170.130" port: 45645 } }
I20260923 20:12:10.175318 15130 leader_election.cc:288] T b90eb6f0659f42ea99ec303527350a9e P ec85741e5ec64d42a20e3c914e696a87 [CANDIDATE]: Term 1 election: Requested vote from peers 711ac19c204c49e5842f923bebce66cc (127.1.170.129:44269), 0bcb955ca73c4fdd8b6b2307210c43ac (127.1.170.130:45645)
I20260923 20:12:10.175848 14779 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b90eb6f0659f42ea99ec303527350a9e" candidate_uuid: "ec85741e5ec64d42a20e3c914e696a87" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "711ac19c204c49e5842f923bebce66cc"
I20260923 20:12:10.176003 14923 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b90eb6f0659f42ea99ec303527350a9e" candidate_uuid: "ec85741e5ec64d42a20e3c914e696a87" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0bcb955ca73c4fdd8b6b2307210c43ac"
I20260923 20:12:10.176054 14779 raft_consensus.cc:3037] T b90eb6f0659f42ea99ec303527350a9e P 711ac19c204c49e5842f923bebce66cc [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:12:10.176255 14923 raft_consensus.cc:3037] T b90eb6f0659f42ea99ec303527350a9e P 0bcb955ca73c4fdd8b6b2307210c43ac [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:12:10.178676 14779 raft_consensus.cc:2454] T b90eb6f0659f42ea99ec303527350a9e P 711ac19c204c49e5842f923bebce66cc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ec85741e5ec64d42a20e3c914e696a87 in term 1.
I20260923 20:12:10.178683 14923 raft_consensus.cc:2454] T b90eb6f0659f42ea99ec303527350a9e P 0bcb955ca73c4fdd8b6b2307210c43ac [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ec85741e5ec64d42a20e3c914e696a87 in term 1.
I20260923 20:12:10.179188 14993 leader_election.cc:302] T b90eb6f0659f42ea99ec303527350a9e P ec85741e5ec64d42a20e3c914e696a87 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 711ac19c204c49e5842f923bebce66cc, ec85741e5ec64d42a20e3c914e696a87; no voters:
I20260923 20:12:10.179499 15130 raft_consensus.cc:2781] T b90eb6f0659f42ea99ec303527350a9e P ec85741e5ec64d42a20e3c914e696a87 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 20:12:10.179935 15130 raft_consensus.cc:686] T b90eb6f0659f42ea99ec303527350a9e P ec85741e5ec64d42a20e3c914e696a87 [term 1 LEADER]: Becoming Leader. State: Replica: ec85741e5ec64d42a20e3c914e696a87, State: Running, Role: LEADER
I20260923 20:12:10.180433 15130 consensus_queue.cc:237] T b90eb6f0659f42ea99ec303527350a9e P ec85741e5ec64d42a20e3c914e696a87 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "711ac19c204c49e5842f923bebce66cc" member_type: VOTER last_known_addr { host: "127.1.170.129" port: 44269 } } peers { permanent_uuid: "ec85741e5ec64d42a20e3c914e696a87" member_type: VOTER last_known_addr { host: "127.1.170.131" port: 34661 } } peers { permanent_uuid: "0bcb955ca73c4fdd8b6b2307210c43ac" member_type: VOTER last_known_addr { host: "127.1.170.130" port: 45645 } }
I20260923 20:12:10.185650 14627 catalog_manager.cc:5495] T b90eb6f0659f42ea99ec303527350a9e P ec85741e5ec64d42a20e3c914e696a87 reported cstate change: term changed from 0 to 1, leader changed from <none> to ec85741e5ec64d42a20e3c914e696a87 (127.1.170.131). New cstate: current_term: 1 leader_uuid: "ec85741e5ec64d42a20e3c914e696a87" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "711ac19c204c49e5842f923bebce66cc" member_type: VOTER last_known_addr { host: "127.1.170.129" port: 44269 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ec85741e5ec64d42a20e3c914e696a87" member_type: VOTER last_known_addr { host: "127.1.170.131" port: 34661 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0bcb955ca73c4fdd8b6b2307210c43ac" member_type: VOTER last_known_addr { host: "127.1.170.130" port: 45645 } health_report { overall_health: UNKNOWN } } }
I20260923 20:12:10.264026 15149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:12:10.257595 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33226 (local address 127.1.170.131:34661)
0923 20:12:10.258188 (+ 593us) server_negotiation.cc:206] Beginning negotiation
0923 20:12:10.258192 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 20:12:10.258210 (+ 18us) server_negotiation.cc:407] Connection header received
0923 20:12:10.258378 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:12:10.258382 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:12:10.258458 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:12:10.258609 (+ 151us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 20:12:10.259256 (+ 647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:12:10.260000 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:12:10.261580 (+ 1580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:12:10.262138 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:12:10.262244 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:12:10.262790 (+ 546us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 20:12:10.263372 (+ 582us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 20:12:10.263568 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0923 20:12:10.263646 (+ 78us) server_negotiation.cc:299] Negotiation successful
0923 20:12:10.263736 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":488,"thread_start_us":354,"threads_started":1}
I20260923 20:12:10.272938 14923 raft_consensus.cc:1261] T b90eb6f0659f42ea99ec303527350a9e P 0bcb955ca73c4fdd8b6b2307210c43ac [term 1 FOLLOWER]: Refusing update from remote peer ec85741e5ec64d42a20e3c914e696a87: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 20:12:10.272979 14779 raft_consensus.cc:1261] T b90eb6f0659f42ea99ec303527350a9e P 711ac19c204c49e5842f923bebce66cc [term 1 FOLLOWER]: Refusing update from remote peer ec85741e5ec64d42a20e3c914e696a87: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 20:12:10.274617 15130 consensus_queue.cc:1035] T b90eb6f0659f42ea99ec303527350a9e P ec85741e5ec64d42a20e3c914e696a87 [LEADER]: Connected to new peer: Peer: permanent_uuid: "711ac19c204c49e5842f923bebce66cc" member_type: VOTER last_known_addr { host: "127.1.170.129" port: 44269 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 20:12:10.275197 15142 consensus_queue.cc:1035] T b90eb6f0659f42ea99ec303527350a9e P ec85741e5ec64d42a20e3c914e696a87 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0bcb955ca73c4fdd8b6b2307210c43ac" member_type: VOTER last_known_addr { host: "127.1.170.130" port: 45645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 20:12:10.288295 15150 mvcc.cc:205] Tried to move back new op lower bound from 7332635976786767872 to 7332635976423333888. Current Snapshot: MvccSnapshot[applied={T|T < 7332635976786767872}]
I20260923 20:12:10.289538 15152 mvcc.cc:205] Tried to move back new op lower bound from 7332635976786767872 to 7332635976423333888. Current Snapshot: MvccSnapshot[applied={T|T < 7332635976786767872}]
I20260923 20:12:10.291914 15151 mvcc.cc:205] Tried to move back new op lower bound from 7332635976786767872 to 7332635976423333888. Current Snapshot: MvccSnapshot[applied={T|T < 7332635976786767872}]
I20260923 20:12:10.302865 14626 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50676:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 20:12:10.303170 14626 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50676:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 20:12:10.307053 14626 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 528212d773b24273adebf10a109a5abe: Sending DeleteTablet for 3 replicas of tablet b90eb6f0659f42ea99ec303527350a9e
I20260923 20:12:10.308449 14758 tablet_service.cc:1514] Processing DeleteTablet for tablet b90eb6f0659f42ea99ec303527350a9e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 20:12:10 UTC) from {username='kudu', principal='kudu/127.1.170.190@KRBTEST.COM'} at 127.0.0.1:49628
I20260923 20:12:10.308494 14903 tablet_service.cc:1514] Processing DeleteTablet for tablet b90eb6f0659f42ea99ec303527350a9e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 20:12:10 UTC) from {username='kudu', principal='kudu/127.1.170.190@KRBTEST.COM'} at 127.0.0.1:58054
I20260923 20:12:10.309408 15037 tablet_service.cc:1514] Processing DeleteTablet for tablet b90eb6f0659f42ea99ec303527350a9e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 20:12:10 UTC) from {username='kudu', principal='kudu/127.1.170.190@KRBTEST.COM'} at 127.0.0.1:33212
I20260923 20:12:10.312340 15161 tablet_replica.cc:330] T b90eb6f0659f42ea99ec303527350a9e P 711ac19c204c49e5842f923bebce66cc: stopping tablet replica
I20260923 20:12:10.312810 15161 raft_consensus.cc:2229] T b90eb6f0659f42ea99ec303527350a9e P 711ac19c204c49e5842f923bebce66cc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 20:12:10.313376 15161 raft_consensus.cc:2258] T b90eb6f0659f42ea99ec303527350a9e P 711ac19c204c49e5842f923bebce66cc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 20:12:10.314349 1706 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskqUuku6/build/asan/bin/kudu pid 14664
I20260923 20:12:10.315549 15161 ts_tablet_manager.cc:1905] T b90eb6f0659f42ea99ec303527350a9e P 711ac19c204c49e5842f923bebce66cc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 20:12:10.323292 15161 ts_tablet_manager.cc:1918] T b90eb6f0659f42ea99ec303527350a9e P 711ac19c204c49e5842f923bebce66cc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 20:12:10.323520 15161 log.cc:1196] T b90eb6f0659f42ea99ec303527350a9e P 711ac19c204c49e5842f923bebce66cc: Deleting WAL directory at /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194121577027-1706-0/minicluster-data/ts-0/wal/wals/b90eb6f0659f42ea99ec303527350a9e
I20260923 20:12:10.324023 15161 ts_tablet_manager.cc:1939] T b90eb6f0659f42ea99ec303527350a9e P 711ac19c204c49e5842f923bebce66cc: Deleting consensus metadata
I20260923 20:12:10.326588 14614 catalog_manager.cc:4841] TS 711ac19c204c49e5842f923bebce66cc (127.1.170.129:44269): tablet b90eb6f0659f42ea99ec303527350a9e (table test-table [id=d24ec6f2e9dc42bfa64852a8f46e46ee]) successfully deleted
I20260923 20:12:10.333838 15119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:12:10.315226 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49638 (local address 127.1.170.129:44269)
0923 20:12:10.315397 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0923 20:12:10.315404 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:12:10.316012 (+ 608us) server_negotiation.cc:407] Connection header received
0923 20:12:10.316298 (+ 286us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:12:10.316304 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:12:10.316395 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:12:10.316478 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:12:10.318686 (+ 2208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:12:10.319758 (+ 1072us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:12:10.321807 (+ 2049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:12:10.322506 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:12:10.322602 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:12:10.327402 (+ 4800us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:12:10.327414 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:12:10.327428 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:12:10.327470 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:12:10.330863 (+ 3393us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:12:10.331646 (+ 783us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:12:10.331654 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:12:10.331657 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:12:10.331790 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:12:10.332191 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:12:10.332198 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:12:10.332201 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:12:10.332576 (+ 375us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:12:10.332903 (+ 327us) server_negotiation.cc:1035] Waiting for connection context
0923 20:12:10.333217 (+ 314us) server_negotiation.cc:299] Negotiation successful
0923 20:12:10.333442 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260923 20:12:10.334671 14819 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:12:10.334867 15162 tablet_replica.cc:330] T b90eb6f0659f42ea99ec303527350a9e P ec85741e5ec64d42a20e3c914e696a87: stopping tablet replica
I20260923 20:12:10.335829 15162 raft_consensus.cc:2229] T b90eb6f0659f42ea99ec303527350a9e P ec85741e5ec64d42a20e3c914e696a87 [term 1 LEADER]: Raft consensus shutting down.
I20260923 20:12:10.336611 15160 tablet_replica.cc:330] T b90eb6f0659f42ea99ec303527350a9e P 0bcb955ca73c4fdd8b6b2307210c43ac: stopping tablet replica
I20260923 20:12:10.337177 15160 raft_consensus.cc:2229] T b90eb6f0659f42ea99ec303527350a9e P 0bcb955ca73c4fdd8b6b2307210c43ac [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 20:12:10.338002 15160 raft_consensus.cc:2258] T b90eb6f0659f42ea99ec303527350a9e P 0bcb955ca73c4fdd8b6b2307210c43ac [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 20:12:10.339141 15162 raft_consensus.cc:2258] T b90eb6f0659f42ea99ec303527350a9e P ec85741e5ec64d42a20e3c914e696a87 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 20:12:10.343022 15160 ts_tablet_manager.cc:1905] T b90eb6f0659f42ea99ec303527350a9e P 0bcb955ca73c4fdd8b6b2307210c43ac: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 20:12:10.350739 15162 ts_tablet_manager.cc:1905] T b90eb6f0659f42ea99ec303527350a9e P ec85741e5ec64d42a20e3c914e696a87: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 20:12:10.351020 15160 ts_tablet_manager.cc:1918] T b90eb6f0659f42ea99ec303527350a9e P 0bcb955ca73c4fdd8b6b2307210c43ac: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 20:12:10.351217 15160 log.cc:1196] T b90eb6f0659f42ea99ec303527350a9e P 0bcb955ca73c4fdd8b6b2307210c43ac: Deleting WAL directory at /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194121577027-1706-0/minicluster-data/ts-1/wal/wals/b90eb6f0659f42ea99ec303527350a9e
I20260923 20:12:10.351792 15160 ts_tablet_manager.cc:1939] T b90eb6f0659f42ea99ec303527350a9e P 0bcb955ca73c4fdd8b6b2307210c43ac: Deleting consensus metadata
I20260923 20:12:10.354801 14611 catalog_manager.cc:4841] TS 0bcb955ca73c4fdd8b6b2307210c43ac (127.1.170.130:45645): tablet b90eb6f0659f42ea99ec303527350a9e (table test-table [id=d24ec6f2e9dc42bfa64852a8f46e46ee]) successfully deleted
I20260923 20:12:10.359272 15162 ts_tablet_manager.cc:1918] T b90eb6f0659f42ea99ec303527350a9e P ec85741e5ec64d42a20e3c914e696a87: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 20:12:10.359495 15162 log.cc:1196] T b90eb6f0659f42ea99ec303527350a9e P ec85741e5ec64d42a20e3c914e696a87: Deleting WAL directory at /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194121577027-1706-0/minicluster-data/ts-2/wal/wals/b90eb6f0659f42ea99ec303527350a9e
I20260923 20:12:10.360051 15162 ts_tablet_manager.cc:1939] T b90eb6f0659f42ea99ec303527350a9e P ec85741e5ec64d42a20e3c914e696a87: Deleting consensus metadata
I20260923 20:12:10.362048 14611 catalog_manager.cc:4841] TS ec85741e5ec64d42a20e3c914e696a87 (127.1.170.131:34661): tablet b90eb6f0659f42ea99ec303527350a9e (table test-table [id=d24ec6f2e9dc42bfa64852a8f46e46ee]) successfully deleted
I20260923 20:12:10.438874 1706 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqUuku6/build/asan/bin/kudu with pid 14664
I20260923 20:12:10.463806 1706 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskqUuku6/build/asan/bin/kudu pid 14828
I20260923 20:12:10.481947 15123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:12:10.464604 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58056 (local address 127.1.170.130:45645)
0923 20:12:10.464790 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0923 20:12:10.464794 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 20:12:10.464991 (+ 197us) server_negotiation.cc:407] Connection header received
0923 20:12:10.465106 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:12:10.465111 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:12:10.465211 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:12:10.465389 (+ 178us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:12:10.467364 (+ 1975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:12:10.468123 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:12:10.470272 (+ 2149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:12:10.470674 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:12:10.470785 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:12:10.474325 (+ 3540us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:12:10.474342 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:12:10.474359 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:12:10.474406 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:12:10.477541 (+ 3135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:12:10.478829 (+ 1288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:12:10.478841 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:12:10.478844 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:12:10.479108 (+ 264us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:12:10.479635 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:12:10.479640 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:12:10.479642 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:12:10.480150 (+ 508us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:12:10.480458 (+ 308us) server_negotiation.cc:1035] Waiting for connection context
0923 20:12:10.480992 (+ 534us) server_negotiation.cc:299] Negotiation successful
0923 20:12:10.481531 (+ 539us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260923 20:12:10.482712 14963 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:12:10.600041 1706 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqUuku6/build/asan/bin/kudu with pid 14828
I20260923 20:12:10.619918 1706 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskqUuku6/build/asan/bin/kudu pid 14972
I20260923 20:12:10.637815 15149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:12:10.620975 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33228 (local address 127.1.170.131:34661)
0923 20:12:10.621204 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0923 20:12:10.621209 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 20:12:10.621233 (+ 24us) server_negotiation.cc:407] Connection header received
0923 20:12:10.621311 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:12:10.621317 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:12:10.621406 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:12:10.621542 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:12:10.623561 (+ 2019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:12:10.624462 (+ 901us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:12:10.626477 (+ 2015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:12:10.627407 (+ 930us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:12:10.627596 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:12:10.630820 (+ 3224us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:12:10.630836 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:12:10.630854 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:12:10.630901 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:12:10.634799 (+ 3898us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:12:10.635586 (+ 787us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:12:10.635594 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:12:10.635596 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:12:10.635727 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:12:10.636143 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:12:10.636147 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:12:10.636149 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:12:10.636507 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:12:10.636858 (+ 351us) server_negotiation.cc:1035] Waiting for connection context
0923 20:12:10.637174 (+ 316us) server_negotiation.cc:299] Negotiation successful
0923 20:12:10.637463 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260923 20:12:10.638659 15097 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:12:10.769049 1706 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqUuku6/build/asan/bin/kudu with pid 14972
I20260923 20:12:10.790834 1706 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskqUuku6/build/asan/bin/kudu pid 14594
I20260923 20:12:10.792209 14657 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:12:10.907356 1706 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqUuku6/build/asan/bin/kudu with pid 14594
I20260923 20:12:10.934255 1706 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260923 20:12:11.270259 1706 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260923 20:12:11.270493 1706 mini_ranger.cc:66] Stopping Ranger...
I20260923 20:12:11.616569 1706 mini_ranger.cc:68] Stopped Ranger
2026-09-23 20:12:11.616 UTC [13646] LOG: received smart shutdown request
2026-09-23 20:12:11.620 UTC [13646] LOG: background worker "logical replication launcher" (PID 13655) exited with exit code 1
2026-09-23 20:12:11.620 UTC [13650] LOG: shutting down
2026-09-23 20:12:11.811 UTC [13646] LOG: database system is shut down
2026-09-23T20:12:11Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (53986 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790194121577027-1706-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 20:12:12 dist-test-slave-th4g krb5kdc[15176](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 20:12:12 dist-test-slave-th4g krb5kdc[15176](info): set up 2 sockets
Sep 23 20:12:12 dist-test-slave-th4g krb5kdc[15176](info): commencing operation
krb5kdc: starting...
W20260923 20:12:13.038683 1706 mini_kdc.cc:121] Time spent starting KDC: real 1.053s user 0.000s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790194121577027-1706-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790194121577027-1706-0/krb5kdc/test-user.keytab.
Sep 23 20:12:13 dist-test-slave-th4g krb5kdc[15176](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194333, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T20:12:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T20:12:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.170.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.170.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.170.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790194121577027-1706-0/krb5kdc/kudu_127.1.170.190.keytab.
Entry for principal kudu/127.1.170.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790194121577027-1706-0/krb5kdc/kudu_127.1.170.190.keytab.
I20260923 20:12:13.279066 1706 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqUuku6/build/asan/bin/kudu
/tmp/dist-test-taskqUuku6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790194121577027-1706-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790194121577027-1706-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790194121577027-1706-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790194121577027-1706-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.170.190:46161
--webserver_interface=127.1.170.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790194121577027-1706-0/krb5kdc/kudu_127.1.170.190.keytab
--principal=kudu/127.1.170.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.170.148:32809
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.170.190:46161,127.1.170.189:46437,127.1.170.188:42949
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790194121577027-1706-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790194121577027-1706-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790194121577027-1706-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:12:13.618820 15189 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:12:13.619131 15189 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:12:13.619278 15189 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:12:13.626668 15189 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 20:12:13.626786 15189 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:12:13.626894 15189 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:12:13.627002 15189 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 20:12:13.627080 15189 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 20:12:13.637440 15189 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.170.148:32809
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790194121577027-1706-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790194121577027-1706-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.170.190:46161,127.1.170.189:46437,127.1.170.188:42949
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790194121577027-1706-0/krb5kdc/kudu_127.1.170.190.keytab
--principal=kudu/127.1.170.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.170.190:46161
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790194121577027-1706-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.170.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790194121577027-1706-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:12:13.638890 15189 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:12:13.640187 15189 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:12:13.652516 15194 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:12:13.652787 15197 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:12:13.652765 15195 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:12:13.653815 15189 server_base.cc:1034] running on GCE node
I20260923 20:12:13.654683 15189 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:12:13.656183 15189 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:12:13.657393 15189 hybrid_clock.cc:648] HybridClock initialized: now 1790194333657335 us; error 57 us; skew 500 ppm
I20260923 20:12:13.657714 15189 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:12:13 dist-test-slave-th4g krb5kdc[15176](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194333, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.170.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:12:13.665864 15189 init.cc:377] Logged in from keytab as kudu/127.1.170.190@KRBTEST.COM (short username kudu)
I20260923 20:12:13.667593 15189 webserver.cc:460] Webserver started at http://127.1.170.190:45737/ using document root <none> and password file <none>
I20260923 20:12:13.668339 15189 fs_manager.cc:359] Metadata directory not provided
I20260923 20:12:13.668504 15189 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:12:13.668820 15189 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:12:13.671968 15189 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790194121577027-1706-0/minicluster-data/master-0/data/instance:
uuid: "deee4a08479d4c0cbcf7273057eca6de"
format_stamp: "Formatted at 2026-09-23 20:12:13 on dist-test-slave-th4g"
server_key: "714232069959f8622b25544ed5c7650b"
server_key_iv: "92457762c60209aeffab2ef805642149"
server_key_version: "encryptionkey@0"
I20260923 20:12:13.672789 15189 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790194121577027-1706-0/minicluster-data/master-0/wal/instance:
uuid: "deee4a08479d4c0cbcf7273057eca6de"
format_stamp: "Formatted at 2026-09-23 20:12:13 on dist-test-slave-th4g"
server_key: "714232069959f8622b25544ed5c7650b"
server_key_iv: "92457762c60209aeffab2ef805642149"
server_key_version: "encryptionkey@0"
I20260923 20:12:13.678185 15189 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260923 20:12:13.681775 15204 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:12:13.683300 15189 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 20:12:13.683502 15189 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790194121577027-1706-0/minicluster-data/master-0/data,/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790194121577027-1706-0/minicluster-data/master-0/wal
uuid: "deee4a08479d4c0cbcf7273057eca6de"
format_stamp: "Formatted at 2026-09-23 20:12:13 on dist-test-slave-th4g"
server_key: "714232069959f8622b25544ed5c7650b"
server_key_iv: "92457762c60209aeffab2ef805642149"
server_key_version: "encryptionkey@0"
I20260923 20:12:13.683730 15189 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790194121577027-1706-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790194121577027-1706-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790194121577027-1706-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:12:13.698562 15189 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:12:13.703011 15189 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:12:13.703305 15189 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:12:13.723465 15189 rpc_server.cc:307] RPC server started. Bound to: 127.1.170.190:46161
I20260923 20:12:13.723511 15255 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.170.190:46161 every 8 connection(s)
I20260923 20:12:13.724888 15189 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790194121577027-1706-0/minicluster-data/master-0/data/info.pb
I20260923 20:12:13.729007 15256 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:12:13.733507 1706 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqUuku6/build/asan/bin/kudu as pid 15189
I20260923 20:12:13.733773 1706 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790194121577027-1706-0/minicluster-data/master-0/wal/instance
I20260923 20:12:13.734705 1706 external_mini_cluster.cc:1428] Setting key 5b68182cb373d248010f7e64ffed4f21
I20260923 20:12:13.738462 15256 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.170.190" port: 46161 } has no permanent_uuid. Determining permanent_uuid...
Sep 23 20:12:13 dist-test-slave-th4g krb5kdc[15176](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194333, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.170.190@KRBTEST.COM for kudu/127.1.170.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.1.170.189@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.170.189@KRBTEST.COM" created.
I20260923 20:12:13.768669 15260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:12:13.743123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33078 (local address 127.1.170.190:46161)
0923 20:12:13.745150 (+ 2027us) server_negotiation.cc:206] Beginning negotiation
0923 20:12:13.745161 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0923 20:12:13.745223 (+ 62us) server_negotiation.cc:407] Connection header received
0923 20:12:13.747221 (+ 1998us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:12:13.747227 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:12:13.747471 (+ 244us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:12:13.747702 (+ 231us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:12:13.749066 (+ 1364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:12:13.750503 (+ 1437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:12:13.752093 (+ 1590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:12:13.752832 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:12:13.753188 (+ 356us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:12:13.758289 (+ 5101us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:12:13.758297 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:12:13.758310 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:12:13.758342 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:12:13.762632 (+ 4290us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:12:13.764306 (+ 1674us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:12:13.764316 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:12:13.764320 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:12:13.764562 (+ 242us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:12:13.765166 (+ 604us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:12:13.765174 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:12:13.765181 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:12:13.766029 (+ 848us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:12:13.766269 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0923 20:12:13.767113 (+ 844us) server_negotiation.cc:299] Negotiation successful
0923 20:12:13.767450 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":27,"server-negotiator.queue_time_us":1784,"spinlock_wait_cycles":27392,"thread_start_us":1520,"threads_started":1}
I20260923 20:12:13.768698 15259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:12:13.743123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.190:46161 (local address 127.0.0.1:33078)
0923 20:12:13.744263 (+ 1140us) negotiation.cc:107] Waiting for socket to connect
0923 20:12:13.744327 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0923 20:12:13.746809 (+ 2482us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:12:13.747722 (+ 913us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:12:13.747730 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:12:13.748502 (+ 772us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:12:13.748853 (+ 351us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:12:13.748871 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:12:13.750684 (+ 1813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:12:13.750693 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:12:13.751937 (+ 1244us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:12:13.751950 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:12:13.753238 (+ 1288us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:12:13.753245 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:12:13.753423 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:12:13.754312 (+ 889us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:12:13.754337 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:12:13.757989 (+ 3652us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:12:13.762844 (+ 4855us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:12:13.762862 (+ 18us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:12:13.762882 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:12:13.764122 (+ 1240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:12:13.764724 (+ 602us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:12:13.764750 (+ 26us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:12:13.764758 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:12:13.765010 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:12:13.766336 (+ 1326us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:12:13.766348 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:12:13.766641 (+ 293us) client_negotiation.cc:769] Sending connection context
0923 20:12:13.766970 (+ 329us) client_negotiation.cc:240] Negotiation successful
0923 20:12:13.767639 (+ 669us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":745,"mutex_wait_us":66,"thread_start_us":450,"threads_started":1}
I20260923 20:12:13.772212 15256 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.170.189" port: 46437 } has no permanent_uuid. Determining permanent_uuid...
I20260923 20:12:13.773710 15259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:12:13.773156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.189:46437 (local address 127.0.0.1:38750)
0923 20:12:13.773383 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0923 20:12:13.773535 (+ 152us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.170.189:46437: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
W20260923 20:12:13.774081 15206 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.170.189:46437: connect: Connection refused (error 111)
I20260923 20:12:13.775934 15259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:12:13.775432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.189:46437 (local address 127.0.0.1:38758)
0923 20:12:13.775637 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0923 20:12:13.775737 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.170.189:46437: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260923 20:12:13.776546 15256 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.170.189:46437: Network error: Client connection negotiation failed: client connection to 127.1.170.189:46437: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.170.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790194121577027-1706-0/krb5kdc/kudu_127.1.170.189.keytab.
Entry for principal kudu/127.1.170.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790194121577027-1706-0/krb5kdc/kudu_127.1.170.189.keytab.
I20260923 20:12:13.794852 1706 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqUuku6/build/asan/bin/kudu
/tmp/dist-test-taskqUuku6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790194121577027-1706-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790194121577027-1706-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790194121577027-1706-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790194121577027-1706-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.170.189:46437
--webserver_interface=127.1.170.189
--webserver_port=0
--keytab_file=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790194121577027-1706-0/krb5kdc/kudu_127.1.170.189.keytab
--principal=kudu/127.1.170.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.170.148:32809
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.170.190:46161,127.1.170.189:46437,127.1.170.188:42949
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790194121577027-1706-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790194121577027-1706-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790194121577027-1706-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260923 20:12:13.826042 15256 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.170.189" port: 46437 } attempt: 1
I20260923 20:12:13.827328 15259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:12:13.826783 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.189:46437 (local address 127.0.0.1:38764)
0923 20:12:13.827014 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0923 20:12:13.827187 (+ 173us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.170.189:46437: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97}
I20260923 20:12:13.828470 15259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:12:13.828076 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.189:46437 (local address 127.0.0.1:38778)
0923 20:12:13.828260 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0923 20:12:13.828335 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.170.189:46437: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
W20260923 20:12:13.828723 15256 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.170.189:46437: Network error: Client connection negotiation failed: client connection to 127.1.170.189:46437: connect: Connection refused (error 111)
I20260923 20:12:13.897023 15256 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.170.189" port: 46437 } attempt: 2
I20260923 20:12:13.898268 15259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:12:13.897752 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.189:46437 (local address 127.0.0.1:38792)
0923 20:12:13.898042 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0923 20:12:13.898129 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.170.189:46437: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20260923 20:12:13.899308 15259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:12:13.898942 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.189:46437 (local address 127.0.0.1:38800)
0923 20:12:13.899090 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0923 20:12:13.899182 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.170.189:46437: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260923 20:12:13.899557 15256 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.170.189:46437: Network error: Client connection negotiation failed: client connection to 127.1.170.189:46437: connect: Connection refused (error 111)
I20260923 20:12:13.990865 15256 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.170.189" port: 46437 } attempt: 3
I20260923 20:12:13.992278 15259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:12:13.991738 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.189:46437 (local address 127.0.0.1:38804)
0923 20:12:13.991979 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0923 20:12:13.992145 (+ 166us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.170.189:46437: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100}
I20260923 20:12:13.993595 15259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:12:13.993164 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.189:46437 (local address 127.0.0.1:38806)
0923 20:12:13.993357 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0923 20:12:13.993430 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.170.189:46437: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
W20260923 20:12:13.993782 15256 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.170.189:46437: Network error: Client connection negotiation failed: client connection to 127.1.170.189:46437: connect: Connection refused (error 111)
W20260923 20:12:14.135401 15263 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:12:14.135794 15263 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:12:14.135994 15263 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260923 20:12:14.137004 15256 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.170.189" port: 46437 } attempt: 4
I20260923 20:12:14.138329 15259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:12:14.137733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.189:46437 (local address 127.0.0.1:38810)
0923 20:12:14.138080 (+ 347us) negotiation.cc:107] Waiting for socket to connect
0923 20:12:14.138199 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.170.189:46437: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":208}
I20260923 20:12:14.139370 15259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:12:14.138991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.189:46437 (local address 127.0.0.1:38812)
0923 20:12:14.139170 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0923 20:12:14.139256 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.170.189:46437: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260923 20:12:14.139551 15256 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.170.189:46437: Network error: Client connection negotiation failed: client connection to 127.1.170.189:46437: connect: Connection refused (error 111)
W20260923 20:12:14.144860 15263 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 20:12:14.145023 15263 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:12:14.145125 15263 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:12:14.145227 15263 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 20:12:14.145303 15263 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 20:12:14.156641 15263 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.170.148:32809
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790194121577027-1706-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790194121577027-1706-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.1.170.190:46161,127.1.170.189:46437,127.1.170.188:42949
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790194121577027-1706-0/krb5kdc/kudu_127.1.170.189.keytab
--principal=kudu/127.1.170.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.170.189:46437
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790194121577027-1706-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.1.170.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790194121577027-1706-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:12:14.158192 15263 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:12:14.159551 15263 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:12:14.171821 15268 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:12:14.171867 15269 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:12:14.171900 15271 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:12:14.173156 15263 server_base.cc:1034] running on GCE node
I20260923 20:12:14.174052 15263 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:12:14.176200 15263 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:12:14.177500 15263 hybrid_clock.cc:648] HybridClock initialized: now 1790194334177441 us; error 76 us; skew 500 ppm
I20260923 20:12:14.178129 15263 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:12:14 dist-test-slave-th4g krb5kdc[15176](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194334, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.170.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:12:14.181836 15263 init.cc:377] Logged in from keytab as kudu/127.1.170.189@KRBTEST.COM (short username kudu)
I20260923 20:12:14.183593 15263 webserver.cc:460] Webserver started at http://127.1.170.189:35683/ using document root <none> and password file <none>
I20260923 20:12:14.184338 15263 fs_manager.cc:359] Metadata directory not provided
I20260923 20:12:14.184445 15263 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:12:14.184929 15263 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:12:14.188274 15263 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790194121577027-1706-0/minicluster-data/master-1/data/instance:
uuid: "d370737315374629a7f6dea33467274c"
format_stamp: "Formatted at 2026-09-23 20:12:14 on dist-test-slave-th4g"
server_key: "71a4fd2fe0b080f3577b285fd1fe8e15"
server_key_iv: "265360e865c1b8dbc04ae30df58e5250"
server_key_version: "encryptionkey@0"
I20260923 20:12:14.189157 15263 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790194121577027-1706-0/minicluster-data/master-1/wal/instance:
uuid: "d370737315374629a7f6dea33467274c"
format_stamp: "Formatted at 2026-09-23 20:12:14 on dist-test-slave-th4g"
server_key: "71a4fd2fe0b080f3577b285fd1fe8e15"
server_key_iv: "265360e865c1b8dbc04ae30df58e5250"
server_key_version: "encryptionkey@0"
I20260923 20:12:14.195783 15263 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260923 20:12:14.199627 15278 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:12:14.201288 15263 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260923 20:12:14.201517 15263 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790194121577027-1706-0/minicluster-data/master-1/data,/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790194121577027-1706-0/minicluster-data/master-1/wal
uuid: "d370737315374629a7f6dea33467274c"
format_stamp: "Formatted at 2026-09-23 20:12:14 on dist-test-slave-th4g"
server_key: "71a4fd2fe0b080f3577b285fd1fe8e15"
server_key_iv: "265360e865c1b8dbc04ae30df58e5250"
server_key_version: "encryptionkey@0"
I20260923 20:12:14.201752 15263 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790194121577027-1706-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790194121577027-1706-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790194121577027-1706-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:12:14.229678 15263 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:12:14.234629 15263 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:12:14.234941 15263 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:12:14.255512 15263 rpc_server.cc:307] RPC server started. Bound to: 127.1.170.189:46437
I20260923 20:12:14.255606 15329 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.170.189:46437 every 8 connection(s)
I20260923 20:12:14.256899 15263 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790194121577027-1706-0/minicluster-data/master-1/data/info.pb
I20260923 20:12:14.258646 1706 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqUuku6/build/asan/bin/kudu as pid 15263
I20260923 20:12:14.258857 1706 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790194121577027-1706-0/minicluster-data/master-1/wal/instance
I20260923 20:12:14.259471 1706 external_mini_cluster.cc:1428] Setting key 5b8ed705ca9aaad97d510275fbd4a43f
I20260923 20:12:14.261912 15330 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:12:14.271642 15330 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.170.190" port: 46161 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.1.170.188@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.170.188@KRBTEST.COM" created.
Sep 23 20:12:14 dist-test-slave-th4g krb5kdc[15176](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194334, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.170.189@KRBTEST.COM for kudu/127.1.170.190@KRBTEST.COM
I20260923 20:12:14.295285 15333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:12:14.274389 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33094 (local address 127.1.170.190:46161)
0923 20:12:14.275002 (+ 613us) server_negotiation.cc:206] Beginning negotiation
0923 20:12:14.275010 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 20:12:14.276052 (+ 1042us) server_negotiation.cc:407] Connection header received
0923 20:12:14.277998 (+ 1946us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:12:14.278006 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:12:14.278109 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:12:14.278250 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:12:14.279780 (+ 1530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:12:14.281037 (+ 1257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:12:14.282905 (+ 1868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:12:14.283313 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:12:14.283422 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:12:14.287315 (+ 3893us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:12:14.287326 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:12:14.287334 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:12:14.287396 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:12:14.291552 (+ 4156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:12:14.292500 (+ 948us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:12:14.292508 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:12:14.292515 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:12:14.292668 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:12:14.293249 (+ 581us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:12:14.293257 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:12:14.293264 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:12:14.293700 (+ 436us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:12:14.293866 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0923 20:12:14.294663 (+ 797us) server_negotiation.cc:299] Negotiation successful
0923 20:12:14.294894 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":441,"thread_start_us":305,"threads_started":1}
I20260923 20:12:14.296620 15334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:12:14.275025 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.190:46161 (local address 127.0.0.1:33094)
0923 20:12:14.275856 (+ 831us) negotiation.cc:107] Waiting for socket to connect
0923 20:12:14.275903 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0923 20:12:14.277675 (+ 1772us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:12:14.278453 (+ 778us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:12:14.278469 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:12:14.278986 (+ 517us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:12:14.279567 (+ 581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:12:14.279584 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:12:14.281239 (+ 1655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:12:14.281244 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:12:14.282749 (+ 1505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:12:14.282762 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:12:14.283550 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:12:14.283555 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:12:14.283868 (+ 313us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:12:14.284554 (+ 686us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:12:14.284581 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:12:14.287008 (+ 2427us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:12:14.291750 (+ 4742us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:12:14.291758 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:12:14.291773 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:12:14.292279 (+ 506us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:12:14.292871 (+ 592us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:12:14.292875 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:12:14.292878 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:12:14.293116 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:12:14.293839 (+ 723us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:12:14.293847 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:12:14.294273 (+ 426us) client_negotiation.cc:769] Sending connection context
0923 20:12:14.294571 (+ 298us) client_negotiation.cc:240] Negotiation successful
0923 20:12:14.295140 (+ 569us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":547,"thread_start_us":355,"threads_started":1}
I20260923 20:12:14.298568 15330 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.170.189" port: 46437 } has no permanent_uuid. Determining permanent_uuid...
Sep 23 20:12:14 dist-test-slave-th4g krb5kdc[15176](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194334, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.170.189@KRBTEST.COM for kudu/127.1.170.189@KRBTEST.COM
I20260923 20:12:14.315935 15334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:12:14.299479 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.189:46437 (local address 127.0.0.1:38816)
0923 20:12:14.299708 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0923 20:12:14.299731 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0923 20:12:14.299840 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:12:14.301519 (+ 1679us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:12:14.301523 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:12:14.302090 (+ 567us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:12:14.302236 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:12:14.302244 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:12:14.303740 (+ 1496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:12:14.303747 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:12:14.304834 (+ 1087us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:12:14.304846 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:12:14.306097 (+ 1251us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:12:14.306102 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:12:14.306428 (+ 326us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:12:14.307142 (+ 714us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:12:14.307172 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:12:14.309741 (+ 2569us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:12:14.313327 (+ 3586us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:12:14.313335 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:12:14.313340 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:12:14.313710 (+ 370us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:12:14.314162 (+ 452us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:12:14.314167 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:12:14.314170 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:12:14.314305 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:12:14.314913 (+ 608us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:12:14.314918 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:12:14.315053 (+ 135us) client_negotiation.cc:769] Sending connection context
0923 20:12:14.315199 (+ 146us) client_negotiation.cc:240] Negotiation successful
0923 20:12:14.315493 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":59,"mutex_wait_us":83}
I20260923 20:12:14.315969 15336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:12:14.300145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38816 (local address 127.1.170.189:46437)
0923 20:12:14.301037 (+ 892us) server_negotiation.cc:206] Beginning negotiation
0923 20:12:14.301043 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:12:14.301070 (+ 27us) server_negotiation.cc:407] Connection header received
0923 20:12:14.301156 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:12:14.301163 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:12:14.301356 (+ 193us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:12:14.301531 (+ 175us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:12:14.302396 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:12:14.303585 (+ 1189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:12:14.304990 (+ 1405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:12:14.305936 (+ 946us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:12:14.306088 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:12:14.309989 (+ 3901us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:12:14.310000 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:12:14.310014 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:12:14.310043 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:12:14.313183 (+ 3140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:12:14.313842 (+ 659us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:12:14.313853 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:12:14.313858 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:12:14.314026 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:12:14.314416 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:12:14.314421 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:12:14.314424 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:12:14.314780 (+ 356us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:12:14.314977 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0923 20:12:14.315338 (+ 361us) server_negotiation.cc:299] Negotiation successful
0923 20:12:14.315606 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":78,"server-negotiator.queue_time_us":734,"thread_start_us":579,"threads_started":1}
I20260923 20:12:14.319180 15330 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.170.188" port: 42949 } has no permanent_uuid. Determining permanent_uuid...
I20260923 20:12:14.321169 15334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:12:14.320511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.188:42949 (local address 127.0.0.1:50876)
0923 20:12:14.320722 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0923 20:12:14.320953 (+ 231us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.170.188:42949: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.170.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790194121577027-1706-0/krb5kdc/kudu_127.1.170.188.keytab.
Entry for principal kudu/127.1.170.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790194121577027-1706-0/krb5kdc/kudu_127.1.170.188.keytab.
W20260923 20:12:14.321529 15279 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.170.188:42949: connect: Connection refused (error 111)
I20260923 20:12:14.323139 1706 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqUuku6/build/asan/bin/kudu
/tmp/dist-test-taskqUuku6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790194121577027-1706-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790194121577027-1706-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790194121577027-1706-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790194121577027-1706-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.170.188:42949
--webserver_interface=127.1.170.188
--webserver_port=0
--keytab_file=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790194121577027-1706-0/krb5kdc/kudu_127.1.170.188.keytab
--principal=kudu/127.1.170.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.170.148:32809
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.170.190:46161,127.1.170.189:46437,127.1.170.188:42949
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790194121577027-1706-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790194121577027-1706-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790194121577027-1706-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260923 20:12:14.323910 15334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:12:14.323486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.188:42949 (local address 127.0.0.1:50890)
0923 20:12:14.323715 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0923 20:12:14.323799 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.170.188:42949: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105}
W20260923 20:12:14.324152 15330 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.170.188:42949: Network error: Client connection negotiation failed: client connection to 127.1.170.188:42949: connect: Connection refused (error 111)
I20260923 20:12:14.373533 15330 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.170.188" port: 42949 } attempt: 1
I20260923 20:12:14.374969 15334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:12:14.374299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.188:42949 (local address 127.0.0.1:50896)
0923 20:12:14.374587 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0923 20:12:14.374740 (+ 153us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.170.188:42949: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97}
I20260923 20:12:14.376600 15334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:12:14.376043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.188:42949 (local address 127.0.0.1:50906)
0923 20:12:14.376301 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0923 20:12:14.376424 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.170.188:42949: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
W20260923 20:12:14.376956 15330 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.170.188:42949: Network error: Client connection negotiation failed: client connection to 127.1.170.188:42949: connect: Connection refused (error 111)
I20260923 20:12:14.438868 15256 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.170.189" port: 46437 } attempt: 5
I20260923 20:12:14.445219 15330 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.170.188" port: 42949 } attempt: 2
I20260923 20:12:14.446507 15334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:12:14.445966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.188:42949 (local address 127.0.0.1:50908)
0923 20:12:14.446189 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0923 20:12:14.446303 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.170.188:42949: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20260923 20:12:14.448148 15334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:12:14.447594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.188:42949 (local address 127.0.0.1:50924)
0923 20:12:14.447829 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0923 20:12:14.447920 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.170.188:42949: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129}
W20260923 20:12:14.448345 15330 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.170.188:42949: Network error: Client connection negotiation failed: client connection to 127.1.170.188:42949: connect: Connection refused (error 111)
Sep 23 20:12:14 dist-test-slave-th4g krb5kdc[15176](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194333, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.170.190@KRBTEST.COM for kudu/127.1.170.189@KRBTEST.COM
I20260923 20:12:14.456483 15259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:12:14.439647 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.189:46437 (local address 127.0.0.1:38818)
0923 20:12:14.439918 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0923 20:12:14.439943 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0923 20:12:14.440091 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:12:14.440481 (+ 390us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:12:14.440486 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:12:14.441027 (+ 541us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:12:14.441316 (+ 289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:12:14.441329 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:12:14.442842 (+ 1513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:12:14.442848 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:12:14.444317 (+ 1469us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:12:14.444337 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:12:14.445565 (+ 1228us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:12:14.445574 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:12:14.446078 (+ 504us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:12:14.446865 (+ 787us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:12:14.446892 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:12:14.449468 (+ 2576us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:12:14.453093 (+ 3625us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:12:14.453100 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:12:14.453104 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:12:14.453510 (+ 406us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:12:14.454021 (+ 511us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:12:14.454027 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:12:14.454035 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:12:14.454174 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:12:14.455150 (+ 976us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:12:14.455157 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:12:14.455379 (+ 222us) client_negotiation.cc:769] Sending connection context
0923 20:12:14.455611 (+ 232us) client_negotiation.cc:240] Negotiation successful
0923 20:12:14.456056 (+ 445us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107}
I20260923 20:12:14.456547 15336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:12:14.439837 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38818 (local address 127.1.170.189:46437)
0923 20:12:14.440024 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0923 20:12:14.440029 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 20:12:14.440050 (+ 21us) server_negotiation.cc:407] Connection header received
0923 20:12:14.440241 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:12:14.440245 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:12:14.440312 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:12:14.440407 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:12:14.441668 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:12:14.442654 (+ 986us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:12:14.444665 (+ 2011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:12:14.445244 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:12:14.445519 (+ 275us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:12:14.449689 (+ 4170us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:12:14.449694 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:12:14.449698 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:12:14.449738 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:12:14.452881 (+ 3143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:12:14.453697 (+ 816us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:12:14.453701 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:12:14.453704 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:12:14.453824 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:12:14.454289 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:12:14.454295 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:12:14.454299 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:12:14.454866 (+ 567us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:12:14.455156 (+ 290us) server_negotiation.cc:1035] Waiting for connection context
0923 20:12:14.455750 (+ 594us) server_negotiation.cc:299] Negotiation successful
0923 20:12:14.456057 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260923 20:12:14.457983 15256 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.170.188" port: 42949 } has no permanent_uuid. Determining permanent_uuid...
I20260923 20:12:14.459335 15259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:12:14.458832 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.188:42949 (local address 127.0.0.1:50934)
0923 20:12:14.459072 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0923 20:12:14.459165 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.170.188:42949: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260923 20:12:14.460412 15259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:12:14.460038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.188:42949 (local address 127.0.0.1:50940)
0923 20:12:14.460207 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0923 20:12:14.460287 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.170.188:42949: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
W20260923 20:12:14.460616 15256 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.170.188:42949: Network error: Client connection negotiation failed: client connection to 127.1.170.188:42949: connect: Connection refused (error 111)
I20260923 20:12:14.511914 15256 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.170.188" port: 42949 } attempt: 1
I20260923 20:12:14.513398 15259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:12:14.512767 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.188:42949 (local address 127.0.0.1:50952)
0923 20:12:14.513024 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0923 20:12:14.513223 (+ 199us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.170.188:42949: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260923 20:12:14.514546 15259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:12:14.514194 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.188:42949 (local address 127.0.0.1:50960)
0923 20:12:14.514347 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0923 20:12:14.514432 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.170.188:42949: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260923 20:12:14.514801 15256 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.170.188:42949: Network error: Client connection negotiation failed: client connection to 127.1.170.188:42949: connect: Connection refused (error 111)
I20260923 20:12:14.539594 15330 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.170.188" port: 42949 } attempt: 3
I20260923 20:12:14.540956 15334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:12:14.540361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.188:42949 (local address 127.0.0.1:50964)
0923 20:12:14.540619 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0923 20:12:14.540740 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.170.188:42949: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
I20260923 20:12:14.542150 15334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:12:14.541659 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.188:42949 (local address 127.0.0.1:50972)
0923 20:12:14.541916 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0923 20:12:14.542003 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.170.188:42949: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":132}
W20260923 20:12:14.542394 15330 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.170.188:42949: Network error: Client connection negotiation failed: client connection to 127.1.170.188:42949: connect: Connection refused (error 111)
I20260923 20:12:14.583048 15256 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.170.188" port: 42949 } attempt: 2
I20260923 20:12:14.584338 15259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:12:14.583778 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.188:42949 (local address 127.0.0.1:50976)
0923 20:12:14.584031 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0923 20:12:14.584142 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.170.188:42949: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260923 20:12:14.585975 15259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:12:14.585456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.188:42949 (local address 127.0.0.1:50978)
0923 20:12:14.585681 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0923 20:12:14.585771 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.170.188:42949: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
W20260923 20:12:14.586297 15256 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.170.188:42949: Network error: Client connection negotiation failed: client connection to 127.1.170.188:42949: connect: Connection refused (error 111)
W20260923 20:12:14.681331 15338 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:12:14.681639 15338 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:12:14.681793 15338 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260923 20:12:14.685662 15330 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.170.188" port: 42949 } attempt: 4
I20260923 20:12:14.686950 15334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:12:14.686359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.188:42949 (local address 127.0.0.1:50990)
0923 20:12:14.686646 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0923 20:12:14.686755 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.170.188:42949: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260923 20:12:14.688071 15334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:12:14.687677 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.188:42949 (local address 127.0.0.1:50992)
0923 20:12:14.687836 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0923 20:12:14.687944 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.170.188:42949: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260923 20:12:14.688294 15330 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.170.188:42949: Network error: Client connection negotiation failed: client connection to 127.1.170.188:42949: connect: Connection refused (error 111)
W20260923 20:12:14.690316 15338 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 20:12:14.690409 15338 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:12:14.690462 15338 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:12:14.690522 15338 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 20:12:14.690567 15338 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 20:12:14.692622 15256 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.170.188" port: 42949 } attempt: 3
I20260923 20:12:14.694262 15259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:12:14.693554 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.188:42949 (local address 127.0.0.1:51006)
0923 20:12:14.693795 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0923 20:12:14.694081 (+ 286us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.170.188:42949: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
I20260923 20:12:14.695607 15259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:12:14.695094 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.188:42949 (local address 127.0.0.1:51010)
0923 20:12:14.695360 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0923 20:12:14.695460 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.170.188:42949: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95}
W20260923 20:12:14.695853 15256 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.170.188:42949: Network error: Client connection negotiation failed: client connection to 127.1.170.188:42949: connect: Connection refused (error 111)
I20260923 20:12:14.701486 15338 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.170.148:32809
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790194121577027-1706-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790194121577027-1706-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.1.170.190:46161,127.1.170.189:46437,127.1.170.188:42949
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790194121577027-1706-0/krb5kdc/kudu_127.1.170.188.keytab
--principal=kudu/127.1.170.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.170.188:42949
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790194121577027-1706-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.1.170.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790194121577027-1706-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:12:14.702884 15338 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:12:14.704151 15338 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:12:14.716547 15343 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:12:14.716511 15346 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:12:14.716497 15344 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:12:14.717461 15338 server_base.cc:1034] running on GCE node
I20260923 20:12:14.718393 15338 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:12:14.719918 15338 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:12:14.721148 15338 hybrid_clock.cc:648] HybridClock initialized: now 1790194334721099 us; error 34 us; skew 500 ppm
I20260923 20:12:14.721482 15338 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:12:14 dist-test-slave-th4g krb5kdc[15176](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194334, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.170.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:12:14.724844 15338 init.cc:377] Logged in from keytab as kudu/127.1.170.188@KRBTEST.COM (short username kudu)
I20260923 20:12:14.726531 15338 webserver.cc:460] Webserver started at http://127.1.170.188:36345/ using document root <none> and password file <none>
I20260923 20:12:14.727255 15338 fs_manager.cc:359] Metadata directory not provided
I20260923 20:12:14.727385 15338 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:12:14.727711 15338 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:12:14.730865 15338 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790194121577027-1706-0/minicluster-data/master-2/data/instance:
uuid: "9ccdafaa162940f58bb43be5ffe29d64"
format_stamp: "Formatted at 2026-09-23 20:12:14 on dist-test-slave-th4g"
server_key: "2ecb77a53bbe3de8f2b32cadb63d5a73"
server_key_iv: "574ee9e42ef19528befa805e8269b391"
server_key_version: "encryptionkey@0"
I20260923 20:12:14.731745 15338 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790194121577027-1706-0/minicluster-data/master-2/wal/instance:
uuid: "9ccdafaa162940f58bb43be5ffe29d64"
format_stamp: "Formatted at 2026-09-23 20:12:14 on dist-test-slave-th4g"
server_key: "2ecb77a53bbe3de8f2b32cadb63d5a73"
server_key_iv: "574ee9e42ef19528befa805e8269b391"
server_key_version: "encryptionkey@0"
I20260923 20:12:14.737339 15338 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260923 20:12:14.741524 15353 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:12:14.743076 15338 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 20:12:14.743268 15338 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790194121577027-1706-0/minicluster-data/master-2/data,/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790194121577027-1706-0/minicluster-data/master-2/wal
uuid: "9ccdafaa162940f58bb43be5ffe29d64"
format_stamp: "Formatted at 2026-09-23 20:12:14 on dist-test-slave-th4g"
server_key: "2ecb77a53bbe3de8f2b32cadb63d5a73"
server_key_iv: "574ee9e42ef19528befa805e8269b391"
server_key_version: "encryptionkey@0"
I20260923 20:12:14.743479 15338 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790194121577027-1706-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790194121577027-1706-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790194121577027-1706-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:12:14.760226 15338 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:12:14.764792 15338 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:12:14.765103 15338 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:12:14.784411 15338 rpc_server.cc:307] RPC server started. Bound to: 127.1.170.188:42949
I20260923 20:12:14.784437 15404 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.170.188:42949 every 8 connection(s)
I20260923 20:12:14.785961 15338 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790194121577027-1706-0/minicluster-data/master-2/data/info.pb
I20260923 20:12:14.789359 1706 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqUuku6/build/asan/bin/kudu as pid 15338
I20260923 20:12:14.789583 1706 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790194121577027-1706-0/minicluster-data/master-2/wal/instance
I20260923 20:12:14.789942 15405 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:12:14.790199 1706 external_mini_cluster.cc:1428] Setting key 04e15d8f119417c2d89906879c177059
I20260923 20:12:14.799902 15405 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.170.190" port: 46161 } has no permanent_uuid. Determining permanent_uuid...
Sep 23 20:12:14 dist-test-slave-th4g krb5kdc[15176](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194333, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.170.188@KRBTEST.COM
Sep 23 20:12:14 dist-test-slave-th4g krb5kdc[15176](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194334, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.170.188@KRBTEST.COM for kudu/127.1.170.190@KRBTEST.COM
I20260923 20:12:14.815073 15408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:12:14.792136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51016 (local address 127.1.170.188:42949)
0923 20:12:14.793021 (+ 885us) server_negotiation.cc:206] Beginning negotiation
0923 20:12:14.793032 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0923 20:12:14.793076 (+ 44us) server_negotiation.cc:407] Connection header received
0923 20:12:14.794847 (+ 1771us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:12:14.794871 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:12:14.795266 (+ 395us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:12:14.795607 (+ 341us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:12:14.797436 (+ 1829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:12:14.798807 (+ 1371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:12:14.800665 (+ 1858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:12:14.801515 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:12:14.801674 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:12:14.806915 (+ 5241us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:12:14.806923 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:12:14.806937 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:12:14.806975 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:12:14.810584 (+ 3609us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:12:14.811434 (+ 850us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:12:14.811444 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:12:14.811454 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:12:14.811680 (+ 226us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:12:14.812135 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:12:14.812140 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:12:14.812143 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:12:14.812734 (+ 591us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:12:14.813037 (+ 303us) server_negotiation.cc:1035] Waiting for connection context
0923 20:12:14.813516 (+ 479us) server_negotiation.cc:299] Negotiation successful
0923 20:12:14.813852 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":660,"thread_start_us":431,"threads_started":1}
I20260923 20:12:14.819949 15410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:12:14.802908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.190:46161 (local address 127.0.0.1:33110)
0923 20:12:14.803862 (+ 954us) negotiation.cc:107] Waiting for socket to connect
0923 20:12:14.803903 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0923 20:12:14.804061 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:12:14.804544 (+ 483us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:12:14.804549 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:12:14.805109 (+ 560us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:12:14.805431 (+ 322us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:12:14.805440 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:12:14.806670 (+ 1230us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:12:14.806678 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:12:14.807865 (+ 1187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:12:14.807877 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:12:14.808625 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:12:14.808631 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:12:14.808802 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:12:14.809425 (+ 623us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:12:14.809441 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:12:14.811935 (+ 2494us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:12:14.816756 (+ 4821us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:12:14.816764 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:12:14.816770 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:12:14.817201 (+ 431us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:12:14.817698 (+ 497us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:12:14.817701 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:12:14.817703 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:12:14.817838 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:12:14.818729 (+ 891us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:12:14.818737 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:12:14.818913 (+ 176us) client_negotiation.cc:769] Sending connection context
0923 20:12:14.819123 (+ 210us) client_negotiation.cc:240] Negotiation successful
0923 20:12:14.819468 (+ 345us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":715,"thread_start_us":560,"threads_started":1}
I20260923 20:12:14.819973 15409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:12:14.802938 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33110 (local address 127.1.170.190:46161)
0923 20:12:14.803503 (+ 565us) server_negotiation.cc:206] Beginning negotiation
0923 20:12:14.803510 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:12:14.804009 (+ 499us) server_negotiation.cc:407] Connection header received
0923 20:12:14.804252 (+ 243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:12:14.804259 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:12:14.804362 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:12:14.804500 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:12:14.805598 (+ 1098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:12:14.806495 (+ 897us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:12:14.808024 (+ 1529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:12:14.808436 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:12:14.808600 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:12:14.812145 (+ 3545us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:12:14.812151 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:12:14.812154 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:12:14.812189 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:12:14.816592 (+ 4403us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:12:14.817340 (+ 748us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:12:14.817347 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:12:14.817350 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:12:14.817549 (+ 199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:12:14.818015 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:12:14.818019 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:12:14.818021 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:12:14.818465 (+ 444us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:12:14.818651 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0923 20:12:14.819257 (+ 606us) server_negotiation.cc:299] Negotiation successful
0923 20:12:14.819544 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":26,"server-negotiator.queue_time_us":415,"thread_start_us":261,"threads_started":1}
I20260923 20:12:14.821486 15405 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.170.189" port: 46437 } has no permanent_uuid. Determining permanent_uuid...
Sep 23 20:12:14 dist-test-slave-th4g krb5kdc[15176](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194333, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.170.190@KRBTEST.COM
Sep 23 20:12:14 dist-test-slave-th4g krb5kdc[15176](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194334, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.170.188@KRBTEST.COM for kudu/127.1.170.189@KRBTEST.COM
I20260923 20:12:14.834478 15411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:12:14.818293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33120 (local address 127.1.170.190:46161)
0923 20:12:14.818908 (+ 615us) server_negotiation.cc:206] Beginning negotiation
0923 20:12:14.818913 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 20:12:14.818931 (+ 18us) server_negotiation.cc:407] Connection header received
0923 20:12:14.818968 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:12:14.818971 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:12:14.819020 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:12:14.819195 (+ 175us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:12:14.820637 (+ 1442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:12:14.821572 (+ 935us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:12:14.823277 (+ 1705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:12:14.823876 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:12:14.823965 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:12:14.828165 (+ 4200us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:12:14.828174 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:12:14.828181 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:12:14.828231 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:12:14.831411 (+ 3180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:12:14.832383 (+ 972us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:12:14.832390 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:12:14.832393 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:12:14.832552 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:12:14.832951 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:12:14.832956 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:12:14.832959 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:12:14.833268 (+ 309us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:12:14.833552 (+ 284us) server_negotiation.cc:1035] Waiting for connection context
0923 20:12:14.833831 (+ 279us) server_negotiation.cc:299] Negotiation successful
0923 20:12:14.834088 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":511,"thread_start_us":324,"threads_started":1}
I20260923 20:12:14.835925 15410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:12:14.822352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.189:46437 (local address 127.0.0.1:38828)
0923 20:12:14.822509 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0923 20:12:14.822527 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0923 20:12:14.822620 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:12:14.822942 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:12:14.822945 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:12:14.823373 (+ 428us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:12:14.823548 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:12:14.823560 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:12:14.824725 (+ 1165us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:12:14.824733 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:12:14.825579 (+ 846us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:12:14.825590 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:12:14.826395 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:12:14.826401 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:12:14.826584 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:12:14.827156 (+ 572us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:12:14.827173 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:12:14.829462 (+ 2289us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:12:14.833354 (+ 3892us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:12:14.833360 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:12:14.833364 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:12:14.833777 (+ 413us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:12:14.834204 (+ 427us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:12:14.834212 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:12:14.834215 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:12:14.834399 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:12:14.834962 (+ 563us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:12:14.834967 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:12:14.835142 (+ 175us) client_negotiation.cc:769] Sending connection context
0923 20:12:14.835289 (+ 147us) client_negotiation.cc:240] Negotiation successful
0923 20:12:14.835576 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51}
I20260923 20:12:14.835958 15336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:12:14.822365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38828 (local address 127.1.170.189:46437)
0923 20:12:14.822549 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0923 20:12:14.822555 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:12:14.822574 (+ 19us) server_negotiation.cc:407] Connection header received
0923 20:12:14.822736 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:12:14.822741 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:12:14.822814 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:12:14.822919 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:12:14.823697 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:12:14.824585 (+ 888us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:12:14.825735 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:12:14.826246 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:12:14.826383 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:12:14.829669 (+ 3286us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:12:14.829680 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:12:14.829683 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:12:14.829718 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:12:14.833189 (+ 3471us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:12:14.833938 (+ 749us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:12:14.833943 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:12:14.833947 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:12:14.834086 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:12:14.834496 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:12:14.834501 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:12:14.834504 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:12:14.834835 (+ 331us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:12:14.834990 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0923 20:12:14.835379 (+ 389us) server_negotiation.cc:299] Negotiation successful
0923 20:12:14.835595 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260923 20:12:14.837042 15405 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.170.188" port: 42949 } has no permanent_uuid. Determining permanent_uuid...
Sep 23 20:12:14 dist-test-slave-th4g krb5kdc[15176](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194334, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.170.188@KRBTEST.COM for kudu/127.1.170.188@KRBTEST.COM
Sep 23 20:12:14 dist-test-slave-th4g krb5kdc[15176](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194333, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.170.189@KRBTEST.COM
I20260923 20:12:14.851282 15408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:12:14.837796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51022 (local address 127.1.170.188:42949)
0923 20:12:14.837970 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0923 20:12:14.837975 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 20:12:14.838098 (+ 123us) server_negotiation.cc:407] Connection header received
0923 20:12:14.838320 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:12:14.838325 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:12:14.838388 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:12:14.838480 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:12:14.839117 (+ 637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:12:14.839770 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:12:14.840873 (+ 1103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:12:14.841382 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:12:14.841512 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:12:14.844922 (+ 3410us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:12:14.844928 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:12:14.844933 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:12:14.844968 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:12:14.848380 (+ 3412us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:12:14.849110 (+ 730us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:12:14.849116 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:12:14.849120 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:12:14.849256 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:12:14.849668 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:12:14.849671 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:12:14.849673 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:12:14.850031 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:12:14.850312 (+ 281us) server_negotiation.cc:1035] Waiting for connection context
0923 20:12:14.850668 (+ 356us) server_negotiation.cc:299] Negotiation successful
0923 20:12:14.850893 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260923 20:12:14.851368 15410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:12:14.837864 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.188:42949 (local address 127.0.0.1:51022)
0923 20:12:14.838012 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0923 20:12:14.838030 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0923 20:12:14.838205 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:12:14.838493 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:12:14.838498 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:12:14.838869 (+ 371us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:12:14.838988 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:12:14.838996 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:12:14.839890 (+ 894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:12:14.839898 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:12:14.840739 (+ 841us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:12:14.840749 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:12:14.841720 (+ 971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:12:14.841727 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:12:14.842001 (+ 274us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:12:14.842579 (+ 578us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:12:14.842595 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:12:14.844754 (+ 2159us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:12:14.848531 (+ 3777us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:12:14.848538 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:12:14.848545 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:12:14.848994 (+ 449us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:12:14.849364 (+ 370us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:12:14.849367 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:12:14.849370 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:12:14.849550 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:12:14.850264 (+ 714us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:12:14.850268 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:12:14.850393 (+ 125us) client_negotiation.cc:769] Sending connection context
0923 20:12:14.850573 (+ 180us) client_negotiation.cc:240] Negotiation successful
0923 20:12:14.851055 (+ 482us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30,"mutex_wait_us":165}
I20260923 20:12:14.852864 15336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:12:14.836358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38834 (local address 127.1.170.189:46437)
0923 20:12:14.836586 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0923 20:12:14.836590 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 20:12:14.836603 (+ 13us) server_negotiation.cc:407] Connection header received
0923 20:12:14.836644 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:12:14.836647 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:12:14.836747 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:12:14.836846 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:12:14.838358 (+ 1512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:12:14.839044 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:12:14.840267 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:12:14.840678 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:12:14.840812 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:12:14.845754 (+ 4942us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:12:14.845763 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:12:14.845769 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:12:14.845810 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:12:14.849341 (+ 3531us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:12:14.850166 (+ 825us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:12:14.850172 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:12:14.850259 (+ 87us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:12:14.850434 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:12:14.850894 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:12:14.850899 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:12:14.850902 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:12:14.851241 (+ 339us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:12:14.851692 (+ 451us) server_negotiation.cc:1035] Waiting for connection context
0923 20:12:14.852266 (+ 574us) server_negotiation.cc:299] Negotiation successful
0923 20:12:14.852466 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132}
I20260923 20:12:14.857453 15405 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9ccdafaa162940f58bb43be5ffe29d64: Bootstrap starting.
I20260923 20:12:14.858345 1706 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 20:12:14.860787 15405 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9ccdafaa162940f58bb43be5ffe29d64: Neither blocks nor log segments found. Creating new log.
I20260923 20:12:14.862114 15405 log.cc:824] T 00000000000000000000000000000000 P 9ccdafaa162940f58bb43be5ffe29d64: Log is configured to *not* fsync() on all Append() calls
I20260923 20:12:14.865371 15405 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9ccdafaa162940f58bb43be5ffe29d64: No bootstrap required, opened a new log
I20260923 20:12:14.873060 15256 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.170.188" port: 42949 } attempt: 4
I20260923 20:12:14.879912 15405 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9ccdafaa162940f58bb43be5ffe29d64 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "deee4a08479d4c0cbcf7273057eca6de" member_type: VOTER last_known_addr { host: "127.1.170.190" port: 46161 } } peers { permanent_uuid: "d370737315374629a7f6dea33467274c" member_type: VOTER last_known_addr { host: "127.1.170.189" port: 46437 } } peers { permanent_uuid: "9ccdafaa162940f58bb43be5ffe29d64" member_type: VOTER last_known_addr { host: "127.1.170.188" port: 42949 } }
I20260923 20:12:14.880424 15405 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9ccdafaa162940f58bb43be5ffe29d64 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:12:14.880745 15405 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9ccdafaa162940f58bb43be5ffe29d64 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ccdafaa162940f58bb43be5ffe29d64, State: Initialized, Role: FOLLOWER
I20260923 20:12:14.881544 15405 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9ccdafaa162940f58bb43be5ffe29d64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "deee4a08479d4c0cbcf7273057eca6de" member_type: VOTER last_known_addr { host: "127.1.170.190" port: 46161 } } peers { permanent_uuid: "d370737315374629a7f6dea33467274c" member_type: VOTER last_known_addr { host: "127.1.170.189" port: 46437 } } peers { permanent_uuid: "9ccdafaa162940f58bb43be5ffe29d64" member_type: VOTER last_known_addr { host: "127.1.170.188" port: 42949 } }
I20260923 20:12:14.889065 15408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:12:14.863919 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51024 (local address 127.1.170.188:42949)
0923 20:12:14.864160 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0923 20:12:14.864168 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 20:12:14.867809 (+ 3641us) server_negotiation.cc:407] Connection header received
0923 20:12:14.867990 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:12:14.867996 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:12:14.868086 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:12:14.868182 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:12:14.870382 (+ 2200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:12:14.871529 (+ 1147us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:12:14.874051 (+ 2522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:12:14.875383 (+ 1332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:12:14.875571 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:12:14.880293 (+ 4722us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:12:14.880300 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:12:14.880305 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:12:14.880345 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:12:14.884977 (+ 4632us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:12:14.886150 (+ 1173us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:12:14.886160 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:12:14.886166 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:12:14.886396 (+ 230us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:12:14.886926 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:12:14.886934 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:12:14.886940 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:12:14.887420 (+ 480us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:12:14.887966 (+ 546us) server_negotiation.cc:1035] Waiting for connection context
0923 20:12:14.888310 (+ 344us) server_negotiation.cc:299] Negotiation successful
0923 20:12:14.888563 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260923 20:12:14.894058 15411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:12:14.863920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33132 (local address 127.1.170.190:46161)
0923 20:12:14.864133 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0923 20:12:14.864141 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 20:12:14.867056 (+ 2915us) server_negotiation.cc:407] Connection header received
0923 20:12:14.867410 (+ 354us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:12:14.867418 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:12:14.867524 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:12:14.867636 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:12:14.870019 (+ 2383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:12:14.871128 (+ 1109us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:12:14.880064 (+ 8936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:12:14.880727 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:12:14.880827 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:12:14.884783 (+ 3956us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:12:14.884793 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:12:14.884799 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:12:14.884854 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:12:14.889317 (+ 4463us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:12:14.890395 (+ 1078us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:12:14.890400 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:12:14.890403 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:12:14.890540 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:12:14.891078 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:12:14.891084 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:12:14.891088 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:12:14.891417 (+ 329us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:12:14.891641 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0923 20:12:14.892191 (+ 550us) server_negotiation.cc:299] Negotiation successful
0923 20:12:14.893742 (+ 1551us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1357,"server-negotiator.queue_time_us":74}
Sep 23 20:12:14 dist-test-slave-th4g krb5kdc[15176](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194333, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.170.190@KRBTEST.COM for kudu/127.1.170.188@KRBTEST.COM
I20260923 20:12:14.898829 15421 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9ccdafaa162940f58bb43be5ffe29d64 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "deee4a08479d4c0cbcf7273057eca6de" member_type: VOTER last_known_addr { host: "127.1.170.190" port: 46161 } } peers { permanent_uuid: "d370737315374629a7f6dea33467274c" member_type: VOTER last_known_addr { host: "127.1.170.189" port: 46437 } } peers { permanent_uuid: "9ccdafaa162940f58bb43be5ffe29d64" member_type: VOTER last_known_addr { host: "127.1.170.188" port: 42949 } } }
I20260923 20:12:14.899271 15421 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9ccdafaa162940f58bb43be5ffe29d64 [sys.catalog]: This master's current role is: FOLLOWER
I20260923 20:12:14.900358 15336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:12:14.863523 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38850 (local address 127.1.170.189:46437)
0923 20:12:14.863751 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0923 20:12:14.863758 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:12:14.871743 (+ 7985us) server_negotiation.cc:407] Connection header received
0923 20:12:14.872253 (+ 510us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:12:14.872259 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:12:14.872343 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:12:14.872408 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:12:14.875545 (+ 3137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:12:14.876632 (+ 1087us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:12:14.878750 (+ 2118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:12:14.879362 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:12:14.879460 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:12:14.892042 (+ 12582us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:12:14.892054 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:12:14.892062 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:12:14.892139 (+ 77us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:12:14.896697 (+ 4558us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:12:14.897856 (+ 1159us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:12:14.897865 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:12:14.897889 (+ 24us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:12:14.898086 (+ 197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:12:14.898613 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:12:14.898617 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:12:14.898620 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:12:14.898982 (+ 362us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:12:14.899220 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0923 20:12:14.899767 (+ 547us) server_negotiation.cc:299] Negotiation successful
0923 20:12:14.900011 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260923 20:12:14.900380 15405 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9ccdafaa162940f58bb43be5ffe29d64 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 20:12:14.906770 15259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:12:14.873632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.188:42949 (local address 127.0.0.1:51036)
0923 20:12:14.874030 (+ 398us) negotiation.cc:107] Waiting for socket to connect
0923 20:12:14.874048 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0923 20:12:14.874155 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:12:14.889695 (+ 15540us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:12:14.889701 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:12:14.890208 (+ 507us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:12:14.890442 (+ 234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:12:14.890454 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:12:14.891990 (+ 1536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:12:14.891998 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:12:14.893372 (+ 1374us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:12:14.893384 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:12:14.895163 (+ 1779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:12:14.895169 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:12:14.895332 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:12:14.896365 (+ 1033us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:12:14.896389 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:12:14.899602 (+ 3213us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:12:14.903513 (+ 3911us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:12:14.903519 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:12:14.903524 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:12:14.903927 (+ 403us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:12:14.904336 (+ 409us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:12:14.904341 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:12:14.904345 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:12:14.904535 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:12:14.905537 (+ 1002us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:12:14.905545 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:12:14.905731 (+ 186us) client_negotiation.cc:769] Sending connection context
0923 20:12:14.905910 (+ 179us) client_negotiation.cc:240] Negotiation successful
0923 20:12:14.906301 (+ 391us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":262}
I20260923 20:12:14.907182 15408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:12:14.874010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51036 (local address 127.1.170.188:42949)
0923 20:12:14.889333 (+ 15323us) server_negotiation.cc:206] Beginning negotiation
0923 20:12:14.889338 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 20:12:14.889358 (+ 20us) server_negotiation.cc:407] Connection header received
0923 20:12:14.889408 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:12:14.889414 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:12:14.889514 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:12:14.890596 (+ 1082us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:12:14.890721 (+ 125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:12:14.891839 (+ 1118us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:12:14.894339 (+ 2500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:12:14.894981 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:12:14.896218 (+ 1237us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:12:14.899811 (+ 3593us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:12:14.899816 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:12:14.899820 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:12:14.899848 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:12:14.903340 (+ 3492us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:12:14.904064 (+ 724us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:12:14.904069 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:12:14.904073 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:12:14.904208 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:12:14.904692 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:12:14.904700 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:12:14.904706 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:12:14.905254 (+ 548us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:12:14.905423 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0923 20:12:14.906144 (+ 721us) server_negotiation.cc:299] Negotiation successful
0923 20:12:14.906425 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15222,"thread_start_us":22363,"threads_started":1}
I20260923 20:12:14.912411 15256 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P deee4a08479d4c0cbcf7273057eca6de: Bootstrap starting.
I20260923 20:12:14.916308 15256 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P deee4a08479d4c0cbcf7273057eca6de: Neither blocks nor log segments found. Creating new log.
I20260923 20:12:14.917402 15256 log.cc:824] T 00000000000000000000000000000000 P deee4a08479d4c0cbcf7273057eca6de: Log is configured to *not* fsync() on all Append() calls
I20260923 20:12:14.920162 15256 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P deee4a08479d4c0cbcf7273057eca6de: No bootstrap required, opened a new log
W20260923 20:12:14.921149 15432 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 9ccdafaa162940f58bb43be5ffe29d64: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260923 20:12:14.921703 15432 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260923 20:12:14.908533 15338 master_runner.cc:186] Error getting master registration for 127.1.170.190:46161: OK, instance_id { permanent_uuid: "deee4a08479d4c0cbcf7273057eca6de" instance_seqno: 1790194333717583 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260923 20:12:14.924564 15338 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260923 20:12:14.932070 15256 raft_consensus.cc:348] T 00000000000000000000000000000000 P deee4a08479d4c0cbcf7273057eca6de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "deee4a08479d4c0cbcf7273057eca6de" member_type: VOTER last_known_addr { host: "127.1.170.190" port: 46161 } } peers { permanent_uuid: "d370737315374629a7f6dea33467274c" member_type: VOTER last_known_addr { host: "127.1.170.189" port: 46437 } } peers { permanent_uuid: "9ccdafaa162940f58bb43be5ffe29d64" member_type: VOTER last_known_addr { host: "127.1.170.188" port: 42949 } }
I20260923 20:12:14.932549 15256 raft_consensus.cc:374] T 00000000000000000000000000000000 P deee4a08479d4c0cbcf7273057eca6de [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:12:14.932673 15256 raft_consensus.cc:729] T 00000000000000000000000000000000 P deee4a08479d4c0cbcf7273057eca6de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: deee4a08479d4c0cbcf7273057eca6de, State: Initialized, Role: FOLLOWER
I20260923 20:12:14.933468 15256 consensus_queue.cc:260] T 00000000000000000000000000000000 P deee4a08479d4c0cbcf7273057eca6de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "deee4a08479d4c0cbcf7273057eca6de" member_type: VOTER last_known_addr { host: "127.1.170.190" port: 46161 } } peers { permanent_uuid: "d370737315374629a7f6dea33467274c" member_type: VOTER last_known_addr { host: "127.1.170.189" port: 46437 } } peers { permanent_uuid: "9ccdafaa162940f58bb43be5ffe29d64" member_type: VOTER last_known_addr { host: "127.1.170.188" port: 42949 } }
I20260923 20:12:14.934621 15435 sys_catalog.cc:455] T 00000000000000000000000000000000 P deee4a08479d4c0cbcf7273057eca6de [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "deee4a08479d4c0cbcf7273057eca6de" member_type: VOTER last_known_addr { host: "127.1.170.190" port: 46161 } } peers { permanent_uuid: "d370737315374629a7f6dea33467274c" member_type: VOTER last_known_addr { host: "127.1.170.189" port: 46437 } } peers { permanent_uuid: "9ccdafaa162940f58bb43be5ffe29d64" member_type: VOTER last_known_addr { host: "127.1.170.188" port: 42949 } } }
I20260923 20:12:14.935140 15435 sys_catalog.cc:458] T 00000000000000000000000000000000 P deee4a08479d4c0cbcf7273057eca6de [sys.catalog]: This master's current role is: FOLLOWER
I20260923 20:12:14.936218 15256 sys_catalog.cc:564] T 00000000000000000000000000000000 P deee4a08479d4c0cbcf7273057eca6de [sys.catalog]: configured and running, proceeding with master startup.
I20260923 20:12:14.945807 15421 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9ccdafaa162940f58bb43be5ffe29d64 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 20:12:14.946178 15421 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9ccdafaa162940f58bb43be5ffe29d64 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "deee4a08479d4c0cbcf7273057eca6de" member_type: VOTER last_known_addr { host: "127.1.170.190" port: 46161 } } peers { permanent_uuid: "d370737315374629a7f6dea33467274c" member_type: VOTER last_known_addr { host: "127.1.170.189" port: 46437 } } peers { permanent_uuid: "9ccdafaa162940f58bb43be5ffe29d64" member_type: VOTER last_known_addr { host: "127.1.170.188" port: 42949 } }
I20260923 20:12:14.947839 15421 leader_election.cc:288] T 00000000000000000000000000000000 P 9ccdafaa162940f58bb43be5ffe29d64 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers deee4a08479d4c0cbcf7273057eca6de (127.1.170.190:46161), d370737315374629a7f6dea33467274c (127.1.170.189:46437)
I20260923 20:12:14.949046 15231 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9ccdafaa162940f58bb43be5ffe29d64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "deee4a08479d4c0cbcf7273057eca6de" is_pre_election: true
I20260923 20:12:14.949672 15231 raft_consensus.cc:2454] T 00000000000000000000000000000000 P deee4a08479d4c0cbcf7273057eca6de [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9ccdafaa162940f58bb43be5ffe29d64 in term 0.
I20260923 20:12:14.950621 15356 leader_election.cc:302] T 00000000000000000000000000000000 P 9ccdafaa162940f58bb43be5ffe29d64 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9ccdafaa162940f58bb43be5ffe29d64, deee4a08479d4c0cbcf7273057eca6de; no voters:
I20260923 20:12:14.951264 15421 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9ccdafaa162940f58bb43be5ffe29d64 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 20:12:14.951412 15421 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9ccdafaa162940f58bb43be5ffe29d64 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 20:12:14.951521 15421 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9ccdafaa162940f58bb43be5ffe29d64 [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:12:14.954968 15421 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9ccdafaa162940f58bb43be5ffe29d64 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "deee4a08479d4c0cbcf7273057eca6de" member_type: VOTER last_known_addr { host: "127.1.170.190" port: 46161 } } peers { permanent_uuid: "d370737315374629a7f6dea33467274c" member_type: VOTER last_known_addr { host: "127.1.170.189" port: 46437 } } peers { permanent_uuid: "9ccdafaa162940f58bb43be5ffe29d64" member_type: VOTER last_known_addr { host: "127.1.170.188" port: 42949 } }
W20260923 20:12:14.957022 15210 tablet.cc:2369] T 00000000000000000000000000000000 P deee4a08479d4c0cbcf7273057eca6de: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 20:12:14.959178 15231 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9ccdafaa162940f58bb43be5ffe29d64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "deee4a08479d4c0cbcf7273057eca6de"
I20260923 20:12:14.959432 15231 raft_consensus.cc:3037] T 00000000000000000000000000000000 P deee4a08479d4c0cbcf7273057eca6de [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:12:14.963068 15231 raft_consensus.cc:2454] T 00000000000000000000000000000000 P deee4a08479d4c0cbcf7273057eca6de [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9ccdafaa162940f58bb43be5ffe29d64 in term 1.
I20260923 20:12:14.963613 15356 leader_election.cc:302] T 00000000000000000000000000000000 P 9ccdafaa162940f58bb43be5ffe29d64 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9ccdafaa162940f58bb43be5ffe29d64, deee4a08479d4c0cbcf7273057eca6de; no voters:
I20260923 20:12:14.967269 15421 leader_election.cc:288] T 00000000000000000000000000000000 P 9ccdafaa162940f58bb43be5ffe29d64 [CANDIDATE]: Term 1 election: Requested vote from peers deee4a08479d4c0cbcf7273057eca6de (127.1.170.190:46161), d370737315374629a7f6dea33467274c (127.1.170.189:46437)
I20260923 20:12:14.967548 15421 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9ccdafaa162940f58bb43be5ffe29d64 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 20:12:14.967724 15421 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9ccdafaa162940f58bb43be5ffe29d64 [term 1 LEADER]: Becoming Leader. State: Replica: 9ccdafaa162940f58bb43be5ffe29d64, State: Running, Role: LEADER
I20260923 20:12:14.968366 15421 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9ccdafaa162940f58bb43be5ffe29d64 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "deee4a08479d4c0cbcf7273057eca6de" member_type: VOTER last_known_addr { host: "127.1.170.190" port: 46161 } } peers { permanent_uuid: "d370737315374629a7f6dea33467274c" member_type: VOTER last_known_addr { host: "127.1.170.189" port: 46437 } } peers { permanent_uuid: "9ccdafaa162940f58bb43be5ffe29d64" member_type: VOTER last_known_addr { host: "127.1.170.188" port: 42949 } }
I20260923 20:12:14.947836 15189 master_runner.cc:186] Error getting master registration for 127.1.170.189:46437: OK, instance_id { permanent_uuid: "d370737315374629a7f6dea33467274c" instance_seqno: 1790194334249765 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260923 20:12:14.970721 15189 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260923 20:12:14.975731 15448 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9ccdafaa162940f58bb43be5ffe29d64 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9ccdafaa162940f58bb43be5ffe29d64. Latest consensus state: current_term: 1 leader_uuid: "9ccdafaa162940f58bb43be5ffe29d64" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "deee4a08479d4c0cbcf7273057eca6de" member_type: VOTER last_known_addr { host: "127.1.170.190" port: 46161 } } peers { permanent_uuid: "d370737315374629a7f6dea33467274c" member_type: VOTER last_known_addr { host: "127.1.170.189" port: 46437 } } peers { permanent_uuid: "9ccdafaa162940f58bb43be5ffe29d64" member_type: VOTER last_known_addr { host: "127.1.170.188" port: 42949 } } }
I20260923 20:12:14.976176 15448 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9ccdafaa162940f58bb43be5ffe29d64 [sys.catalog]: This master's current role is: LEADER
W20260923 20:12:14.976455 15446 catalog_manager.cc:1558] T 00000000000000000000000000000000 P deee4a08479d4c0cbcf7273057eca6de: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260923 20:12:14.976588 15446 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260923 20:12:14.977219 15450 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 20:12:14.979779 15450 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 20:12:14.987534 15330 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.170.188" port: 42949 } attempt: 5
I20260923 20:12:14.992664 15231 raft_consensus.cc:1261] T 00000000000000000000000000000000 P deee4a08479d4c0cbcf7273057eca6de [term 1 FOLLOWER]: Refusing update from remote peer 9ccdafaa162940f58bb43be5ffe29d64: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 20:12:14.993928 15448 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 9ccdafaa162940f58bb43be5ffe29d64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "deee4a08479d4c0cbcf7273057eca6de" member_type: VOTER last_known_addr { host: "127.1.170.190" port: 46161 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Sep 23 20:12:14 dist-test-slave-th4g krb5kdc[15176](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194334, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.170.189@KRBTEST.COM for kudu/127.1.170.188@KRBTEST.COM
I20260923 20:12:15.016306 15334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:12:14.988132 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.188:42949 (local address 127.0.0.1:51042)
0923 20:12:14.988487 (+ 355us) negotiation.cc:107] Waiting for socket to connect
0923 20:12:14.988508 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0923 20:12:14.988616 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:12:14.989545 (+ 929us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:12:14.989551 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:12:14.992226 (+ 2675us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:12:14.992444 (+ 218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:12:14.992458 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:12:14.994422 (+ 1964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:12:14.994428 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:12:14.995898 (+ 1470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:12:14.995917 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:12:14.997200 (+ 1283us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:12:14.997207 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:12:14.997467 (+ 260us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:12:14.998296 (+ 829us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:12:14.998346 (+ 50us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:12:15.002312 (+ 3966us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:12:15.010620 (+ 8308us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:12:15.010649 (+ 29us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:12:15.010654 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:12:15.011642 (+ 988us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:12:15.012338 (+ 696us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:12:15.012366 (+ 28us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:12:15.012370 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:12:15.012638 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:12:15.013903 (+ 1265us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:12:15.013908 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:12:15.014184 (+ 276us) client_negotiation.cc:769] Sending connection context
0923 20:12:15.014423 (+ 239us) client_negotiation.cc:240] Negotiation successful
0923 20:12:15.014990 (+ 567us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":237}
W20260923 20:12:15.017266 15359 tablet.cc:2369] T 00000000000000000000000000000000 P 9ccdafaa162940f58bb43be5ffe29d64: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 20:12:15.018038 15408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:12:14.988820 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51042 (local address 127.1.170.188:42949)
0923 20:12:14.989082 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0923 20:12:14.989088 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:12:14.989106 (+ 18us) server_negotiation.cc:407] Connection header received
0923 20:12:14.989159 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:12:14.989165 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:12:14.989241 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:12:14.989337 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:12:14.992655 (+ 3318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:12:14.993851 (+ 1196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:12:14.996118 (+ 2267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:12:14.996814 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:12:14.996949 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:12:15.002592 (+ 5643us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:12:15.002599 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:12:15.002604 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:12:15.002645 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:12:15.010395 (+ 7750us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:12:15.011903 (+ 1508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:12:15.011909 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:12:15.011913 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:12:15.012178 (+ 265us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:12:15.012914 (+ 736us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:12:15.012919 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:12:15.012923 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:12:15.013675 (+ 752us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:12:15.015295 (+ 1620us) server_negotiation.cc:1035] Waiting for connection context
0923 20:12:15.015708 (+ 413us) server_negotiation.cc:299] Negotiation successful
0923 20:12:15.017195 (+ 1487us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":155}
I20260923 20:12:15.024417 15330 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d370737315374629a7f6dea33467274c: Bootstrap starting.
I20260923 20:12:15.029093 15330 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d370737315374629a7f6dea33467274c: Neither blocks nor log segments found. Creating new log.
I20260923 20:12:15.031471 15330 log.cc:824] T 00000000000000000000000000000000 P d370737315374629a7f6dea33467274c: Log is configured to *not* fsync() on all Append() calls
I20260923 20:12:15.044664 15330 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d370737315374629a7f6dea33467274c: No bootstrap required, opened a new log
I20260923 20:12:15.064596 15435 sys_catalog.cc:455] T 00000000000000000000000000000000 P deee4a08479d4c0cbcf7273057eca6de [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9ccdafaa162940f58bb43be5ffe29d64. Latest consensus state: current_term: 1 leader_uuid: "9ccdafaa162940f58bb43be5ffe29d64" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "deee4a08479d4c0cbcf7273057eca6de" member_type: VOTER last_known_addr { host: "127.1.170.190" port: 46161 } } peers { permanent_uuid: "d370737315374629a7f6dea33467274c" member_type: VOTER last_known_addr { host: "127.1.170.189" port: 46437 } } peers { permanent_uuid: "9ccdafaa162940f58bb43be5ffe29d64" member_type: VOTER last_known_addr { host: "127.1.170.188" port: 42949 } } }
I20260923 20:12:15.064960 15435 sys_catalog.cc:458] T 00000000000000000000000000000000 P deee4a08479d4c0cbcf7273057eca6de [sys.catalog]: This master's current role is: FOLLOWER
I20260923 20:12:15.068348 15452 mvcc.cc:205] Tried to move back new op lower bound from 7332635996111208448 to 7332635996039901184. Current Snapshot: MvccSnapshot[applied={T|T < 7332635996111208448}]
I20260923 20:12:15.069079 15435 sys_catalog.cc:455] T 00000000000000000000000000000000 P deee4a08479d4c0cbcf7273057eca6de [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9ccdafaa162940f58bb43be5ffe29d64" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "deee4a08479d4c0cbcf7273057eca6de" member_type: VOTER last_known_addr { host: "127.1.170.190" port: 46161 } } peers { permanent_uuid: "d370737315374629a7f6dea33467274c" member_type: VOTER last_known_addr { host: "127.1.170.189" port: 46437 } } peers { permanent_uuid: "9ccdafaa162940f58bb43be5ffe29d64" member_type: VOTER last_known_addr { host: "127.1.170.188" port: 42949 } } }
I20260923 20:12:15.069335 15435 sys_catalog.cc:458] T 00000000000000000000000000000000 P deee4a08479d4c0cbcf7273057eca6de [sys.catalog]: This master's current role is: FOLLOWER
I20260923 20:12:15.075851 15330 raft_consensus.cc:348] T 00000000000000000000000000000000 P d370737315374629a7f6dea33467274c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "deee4a08479d4c0cbcf7273057eca6de" member_type: VOTER last_known_addr { host: "127.1.170.190" port: 46161 } } peers { permanent_uuid: "d370737315374629a7f6dea33467274c" member_type: VOTER last_known_addr { host: "127.1.170.189" port: 46437 } } peers { permanent_uuid: "9ccdafaa162940f58bb43be5ffe29d64" member_type: VOTER last_known_addr { host: "127.1.170.188" port: 42949 } }
I20260923 20:12:15.077102 15330 raft_consensus.cc:374] T 00000000000000000000000000000000 P d370737315374629a7f6dea33467274c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:12:15.079080 15330 raft_consensus.cc:729] T 00000000000000000000000000000000 P d370737315374629a7f6dea33467274c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d370737315374629a7f6dea33467274c, State: Initialized, Role: FOLLOWER
I20260923 20:12:15.080071 15330 consensus_queue.cc:260] T 00000000000000000000000000000000 P d370737315374629a7f6dea33467274c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "deee4a08479d4c0cbcf7273057eca6de" member_type: VOTER last_known_addr { host: "127.1.170.190" port: 46161 } } peers { permanent_uuid: "d370737315374629a7f6dea33467274c" member_type: VOTER last_known_addr { host: "127.1.170.189" port: 46437 } } peers { permanent_uuid: "9ccdafaa162940f58bb43be5ffe29d64" member_type: VOTER last_known_addr { host: "127.1.170.188" port: 42949 } }
I20260923 20:12:15.089087 15450 catalog_manager.cc:1347] Generated new cluster ID: 9f771f7e5c1b44849062c72607808663
I20260923 20:12:15.089218 15450 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 20:12:15.090312 15448 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9ccdafaa162940f58bb43be5ffe29d64 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9ccdafaa162940f58bb43be5ffe29d64" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "deee4a08479d4c0cbcf7273057eca6de" member_type: VOTER last_known_addr { host: "127.1.170.190" port: 46161 } } peers { permanent_uuid: "d370737315374629a7f6dea33467274c" member_type: VOTER last_known_addr { host: "127.1.170.189" port: 46437 } } peers { permanent_uuid: "9ccdafaa162940f58bb43be5ffe29d64" member_type: VOTER last_known_addr { host: "127.1.170.188" port: 42949 } } }
I20260923 20:12:15.090596 15448 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9ccdafaa162940f58bb43be5ffe29d64 [sys.catalog]: This master's current role is: LEADER
I20260923 20:12:15.091377 15457 sys_catalog.cc:455] T 00000000000000000000000000000000 P d370737315374629a7f6dea33467274c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "deee4a08479d4c0cbcf7273057eca6de" member_type: VOTER last_known_addr { host: "127.1.170.190" port: 46161 } } peers { permanent_uuid: "d370737315374629a7f6dea33467274c" member_type: VOTER last_known_addr { host: "127.1.170.189" port: 46437 } } peers { permanent_uuid: "9ccdafaa162940f58bb43be5ffe29d64" member_type: VOTER last_known_addr { host: "127.1.170.188" port: 42949 } } }
I20260923 20:12:15.091872 15457 sys_catalog.cc:458] T 00000000000000000000000000000000 P d370737315374629a7f6dea33467274c [sys.catalog]: This master's current role is: FOLLOWER
I20260923 20:12:15.093786 15330 sys_catalog.cc:564] T 00000000000000000000000000000000 P d370737315374629a7f6dea33467274c [sys.catalog]: configured and running, proceeding with master startup.
I20260923 20:12:15.094522 15305 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9ccdafaa162940f58bb43be5ffe29d64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d370737315374629a7f6dea33467274c" is_pre_election: true
I20260923 20:12:15.095185 15305 raft_consensus.cc:2454] T 00000000000000000000000000000000 P d370737315374629a7f6dea33467274c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9ccdafaa162940f58bb43be5ffe29d64 in term 0.
I20260923 20:12:15.096164 15304 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9ccdafaa162940f58bb43be5ffe29d64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d370737315374629a7f6dea33467274c"
I20260923 20:12:15.096421 15304 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d370737315374629a7f6dea33467274c [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:12:15.101596 15304 raft_consensus.cc:2454] T 00000000000000000000000000000000 P d370737315374629a7f6dea33467274c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9ccdafaa162940f58bb43be5ffe29d64 in term 1.
I20260923 20:12:15.102423 15303 raft_consensus.cc:1261] T 00000000000000000000000000000000 P d370737315374629a7f6dea33467274c [term 1 FOLLOWER]: Refusing update from remote peer 9ccdafaa162940f58bb43be5ffe29d64: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 20:12:15.104333 15421 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 9ccdafaa162940f58bb43be5ffe29d64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d370737315374629a7f6dea33467274c" member_type: VOTER last_known_addr { host: "127.1.170.189" port: 46437 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 20:12:15.128357 15450 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 20:12:15.130308 15450 catalog_manager.cc:1504] Loading token signing keys...
I20260923 20:12:15.152930 15450 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9ccdafaa162940f58bb43be5ffe29d64: Generated new TSK 0
I20260923 20:12:15.153745 15457 sys_catalog.cc:455] T 00000000000000000000000000000000 P d370737315374629a7f6dea33467274c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9ccdafaa162940f58bb43be5ffe29d64. Latest consensus state: current_term: 1 leader_uuid: "9ccdafaa162940f58bb43be5ffe29d64" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "deee4a08479d4c0cbcf7273057eca6de" member_type: VOTER last_known_addr { host: "127.1.170.190" port: 46161 } } peers { permanent_uuid: "d370737315374629a7f6dea33467274c" member_type: VOTER last_known_addr { host: "127.1.170.189" port: 46437 } } peers { permanent_uuid: "9ccdafaa162940f58bb43be5ffe29d64" member_type: VOTER last_known_addr { host: "127.1.170.188" port: 42949 } } }
I20260923 20:12:15.154004 15450 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 20:12:15.154074 15457 sys_catalog.cc:458] T 00000000000000000000000000000000 P d370737315374629a7f6dea33467274c [sys.catalog]: This master's current role is: FOLLOWER
I20260923 20:12:15.154458 15466 sys_catalog.cc:455] T 00000000000000000000000000000000 P d370737315374629a7f6dea33467274c [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9ccdafaa162940f58bb43be5ffe29d64" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "deee4a08479d4c0cbcf7273057eca6de" member_type: VOTER last_known_addr { host: "127.1.170.190" port: 46161 } } peers { permanent_uuid: "d370737315374629a7f6dea33467274c" member_type: VOTER last_known_addr { host: "127.1.170.189" port: 46437 } } peers { permanent_uuid: "9ccdafaa162940f58bb43be5ffe29d64" member_type: VOTER last_known_addr { host: "127.1.170.188" port: 42949 } } }
I20260923 20:12:15.155699 15448 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9ccdafaa162940f58bb43be5ffe29d64 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9ccdafaa162940f58bb43be5ffe29d64" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "deee4a08479d4c0cbcf7273057eca6de" member_type: VOTER last_known_addr { host: "127.1.170.190" port: 46161 } } peers { permanent_uuid: "d370737315374629a7f6dea33467274c" member_type: VOTER last_known_addr { host: "127.1.170.189" port: 46437 } } peers { permanent_uuid: "9ccdafaa162940f58bb43be5ffe29d64" member_type: VOTER last_known_addr { host: "127.1.170.188" port: 42949 } } }
I20260923 20:12:15.155962 15448 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9ccdafaa162940f58bb43be5ffe29d64 [sys.catalog]: This master's current role is: LEADER
I20260923 20:12:15.162360 15466 sys_catalog.cc:458] T 00000000000000000000000000000000 P d370737315374629a7f6dea33467274c [sys.catalog]: This master's current role is: FOLLOWER
I20260923 20:12:15.189860 15474 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260923 20:12:15.978626 15446 catalog_manager.cc:1259] Loaded cluster ID: 9f771f7e5c1b44849062c72607808663
I20260923 20:12:15.978765 15446 catalog_manager.cc:1552] T 00000000000000000000000000000000 P deee4a08479d4c0cbcf7273057eca6de: loading cluster ID for follower catalog manager: success
I20260923 20:12:15.981201 15446 catalog_manager.cc:1574] T 00000000000000000000000000000000 P deee4a08479d4c0cbcf7273057eca6de: acquiring CA information for follower catalog manager: success
I20260923 20:12:15.983003 15446 catalog_manager.cc:1602] T 00000000000000000000000000000000 P deee4a08479d4c0cbcf7273057eca6de: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260923 20:12:16.194291 15474 catalog_manager.cc:1259] Loaded cluster ID: 9f771f7e5c1b44849062c72607808663
I20260923 20:12:16.194396 15474 catalog_manager.cc:1552] T 00000000000000000000000000000000 P d370737315374629a7f6dea33467274c: loading cluster ID for follower catalog manager: success
I20260923 20:12:16.197003 15474 catalog_manager.cc:1574] T 00000000000000000000000000000000 P d370737315374629a7f6dea33467274c: acquiring CA information for follower catalog manager: success
I20260923 20:12:16.198892 15474 catalog_manager.cc:1602] T 00000000000000000000000000000000 P d370737315374629a7f6dea33467274c: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260923 20:12:16.268504 1706 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskqUuku6/build/asan/bin/kudu pid 15189
I20260923 20:12:16.269788 15251 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:12:16.399068 1706 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqUuku6/build/asan/bin/kudu with pid 15189
I20260923 20:12:16.420671 1706 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskqUuku6/build/asan/bin/kudu pid 15263
I20260923 20:12:16.421841 15325 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:12:16.531834 1706 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqUuku6/build/asan/bin/kudu with pid 15263
I20260923 20:12:16.550443 1706 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskqUuku6/build/asan/bin/kudu pid 15338
I20260923 20:12:16.551523 15400 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:12:16.659080 1706 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqUuku6/build/asan/bin/kudu with pid 15338
2026-09-23T20:12:16Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (4715 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790194121577027-1706-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 20:12:16 dist-test-slave-th4g krb5kdc[15484](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 20:12:16 dist-test-slave-th4g krb5kdc[15484](info): set up 2 sockets
Sep 23 20:12:16 dist-test-slave-th4g krb5kdc[15484](info): commencing operation
krb5kdc: starting...
W20260923 20:12:17.803584 1706 mini_kdc.cc:121] Time spent starting KDC: real 1.103s user 0.004s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790194121577027-1706-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790194121577027-1706-0/krb5kdc/test-user.keytab.
Sep 23 20:12:17 dist-test-slave-th4g krb5kdc[15484](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194337, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T20:12:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T20:12:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.170.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.170.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.170.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790194121577027-1706-0/krb5kdc/kudu_127.1.170.190.keytab.
Entry for principal kudu/127.1.170.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790194121577027-1706-0/krb5kdc/kudu_127.1.170.190.keytab.
I20260923 20:12:18.061935 1706 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqUuku6/build/asan/bin/kudu
/tmp/dist-test-taskqUuku6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790194121577027-1706-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790194121577027-1706-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790194121577027-1706-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790194121577027-1706-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.170.190:43771
--webserver_interface=127.1.170.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790194121577027-1706-0/krb5kdc/kudu_127.1.170.190.keytab
--principal=kudu/127.1.170.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.170.148:46357
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.170.190:43771
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790194121577027-1706-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790194121577027-1706-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790194121577027-1706-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:12:18.402199 15497 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:12:18.402499 15497 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:12:18.402680 15497 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:12:18.410096 15497 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 20:12:18.410233 15497 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:12:18.410320 15497 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:12:18.410414 15497 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 20:12:18.410465 15497 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 20:12:18.420691 15497 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.170.148:46357
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790194121577027-1706-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790194121577027-1706-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.170.190:43771
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790194121577027-1706-0/krb5kdc/kudu_127.1.170.190.keytab
--principal=kudu/127.1.170.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.170.190:43771
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790194121577027-1706-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.170.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790194121577027-1706-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:12:18.422122 15497 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:12:18.423477 15497 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:12:18.436586 15502 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:12:18.436544 15503 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:12:18.436651 15505 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:12:18.438295 15497 server_base.cc:1034] running on GCE node
I20260923 20:12:18.439177 15497 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:12:18.440836 15497 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:12:18.442029 15497 hybrid_clock.cc:648] HybridClock initialized: now 1790194338441942 us; error 60 us; skew 500 ppm
I20260923 20:12:18.442435 15497 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:12:18 dist-test-slave-th4g krb5kdc[15484](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194338, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.170.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:12:18.445873 15497 init.cc:377] Logged in from keytab as kudu/127.1.170.190@KRBTEST.COM (short username kudu)
I20260923 20:12:18.447758 15497 webserver.cc:460] Webserver started at http://127.1.170.190:42971/ using document root <none> and password file <none>
I20260923 20:12:18.448508 15497 fs_manager.cc:359] Metadata directory not provided
I20260923 20:12:18.448647 15497 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:12:18.449002 15497 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:12:18.452286 15497 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790194121577027-1706-0/minicluster-data/master-0/data/instance:
uuid: "2d0baa738a1b4dbcaefede5b58ba3d1f"
format_stamp: "Formatted at 2026-09-23 20:12:18 on dist-test-slave-th4g"
server_key: "c004f3581a5449fb4c4e171a1afb9828"
server_key_iv: "14f4704fa238ea9d2f0a3d0c3a8123f9"
server_key_version: "encryptionkey@0"
I20260923 20:12:18.453161 15497 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790194121577027-1706-0/minicluster-data/master-0/wal/instance:
uuid: "2d0baa738a1b4dbcaefede5b58ba3d1f"
format_stamp: "Formatted at 2026-09-23 20:12:18 on dist-test-slave-th4g"
server_key: "c004f3581a5449fb4c4e171a1afb9828"
server_key_iv: "14f4704fa238ea9d2f0a3d0c3a8123f9"
server_key_version: "encryptionkey@0"
I20260923 20:12:18.459411 15497 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260923 20:12:18.463267 15512 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:12:18.465128 15497 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260923 20:12:18.465365 15497 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790194121577027-1706-0/minicluster-data/master-0/data,/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790194121577027-1706-0/minicluster-data/master-0/wal
uuid: "2d0baa738a1b4dbcaefede5b58ba3d1f"
format_stamp: "Formatted at 2026-09-23 20:12:18 on dist-test-slave-th4g"
server_key: "c004f3581a5449fb4c4e171a1afb9828"
server_key_iv: "14f4704fa238ea9d2f0a3d0c3a8123f9"
server_key_version: "encryptionkey@0"
I20260923 20:12:18.465593 15497 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790194121577027-1706-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790194121577027-1706-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790194121577027-1706-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:12:18.486829 15497 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:12:18.492220 15497 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:12:18.492550 15497 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:12:18.513372 15497 rpc_server.cc:307] RPC server started. Bound to: 127.1.170.190:43771
I20260923 20:12:18.513471 15563 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.170.190:43771 every 8 connection(s)
I20260923 20:12:18.514919 15497 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790194121577027-1706-0/minicluster-data/master-0/data/info.pb
I20260923 20:12:18.516623 1706 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqUuku6/build/asan/bin/kudu as pid 15497
I20260923 20:12:18.516868 1706 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790194121577027-1706-0/minicluster-data/master-0/wal/instance
I20260923 20:12:18.517598 1706 external_mini_cluster.cc:1428] Setting key ea2ed972307e63d166643d3030d1b202
I20260923 20:12:18.519340 15564 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:12:18.533128 15564 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2d0baa738a1b4dbcaefede5b58ba3d1f: Bootstrap starting.
Sep 23 20:12:18 dist-test-slave-th4g krb5kdc[15484](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194337, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.170.190@KRBTEST.COM
I20260923 20:12:18.537228 15564 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2d0baa738a1b4dbcaefede5b58ba3d1f: Neither blocks nor log segments found. Creating new log.
I20260923 20:12:18.538669 15564 log.cc:824] T 00000000000000000000000000000000 P 2d0baa738a1b4dbcaefede5b58ba3d1f: Log is configured to *not* fsync() on all Append() calls
I20260923 20:12:18.541805 15564 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2d0baa738a1b4dbcaefede5b58ba3d1f: No bootstrap required, opened a new log
I20260923 20:12:18.543202 15567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:12:18.519574 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42574 (local address 127.1.170.190:43771)
0923 20:12:18.520397 (+ 823us) server_negotiation.cc:206] Beginning negotiation
0923 20:12:18.520411 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0923 20:12:18.520460 (+ 49us) server_negotiation.cc:407] Connection header received
0923 20:12:18.522619 (+ 2159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:12:18.522649 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:12:18.523201 (+ 552us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:12:18.523632 (+ 431us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:12:18.525115 (+ 1483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:12:18.527190 (+ 2075us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:12:18.528625 (+ 1435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:12:18.529687 (+ 1062us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:12:18.529894 (+ 207us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:12:18.534466 (+ 4572us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:12:18.534478 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:12:18.534500 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:12:18.534547 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:12:18.538729 (+ 4182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:12:18.539617 (+ 888us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:12:18.539629 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:12:18.539650 (+ 21us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:12:18.539877 (+ 227us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:12:18.540355 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:12:18.540361 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:12:18.540366 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:12:18.540938 (+ 572us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:12:18.541263 (+ 325us) server_negotiation.cc:1035] Waiting for connection context
0923 20:12:18.541760 (+ 497us) server_negotiation.cc:299] Negotiation successful
0923 20:12:18.542207 (+ 447us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":570,"thread_start_us":334,"threads_started":1}
I20260923 20:12:18.547935 1706 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 20:12:18.551486 15564 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2d0baa738a1b4dbcaefede5b58ba3d1f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d0baa738a1b4dbcaefede5b58ba3d1f" member_type: VOTER last_known_addr { host: "127.1.170.190" port: 43771 } }
I20260923 20:12:18.551908 15564 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2d0baa738a1b4dbcaefede5b58ba3d1f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:12:18.552016 15564 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2d0baa738a1b4dbcaefede5b58ba3d1f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d0baa738a1b4dbcaefede5b58ba3d1f, State: Initialized, Role: FOLLOWER
I20260923 20:12:18.552639 15564 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2d0baa738a1b4dbcaefede5b58ba3d1f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d0baa738a1b4dbcaefede5b58ba3d1f" member_type: VOTER last_known_addr { host: "127.1.170.190" port: 43771 } }
I20260923 20:12:18.552892 15564 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2d0baa738a1b4dbcaefede5b58ba3d1f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 20:12:18.553066 15564 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2d0baa738a1b4dbcaefede5b58ba3d1f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 20:12:18.553238 15564 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2d0baa738a1b4dbcaefede5b58ba3d1f [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:12:18.555563 15564 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2d0baa738a1b4dbcaefede5b58ba3d1f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d0baa738a1b4dbcaefede5b58ba3d1f" member_type: VOTER last_known_addr { host: "127.1.170.190" port: 43771 } }
I20260923 20:12:18.556102 15564 leader_election.cc:302] T 00000000000000000000000000000000 P 2d0baa738a1b4dbcaefede5b58ba3d1f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2d0baa738a1b4dbcaefede5b58ba3d1f; no voters:
I20260923 20:12:18.556742 15564 leader_election.cc:288] T 00000000000000000000000000000000 P 2d0baa738a1b4dbcaefede5b58ba3d1f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 20:12:18.556901 15569 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2d0baa738a1b4dbcaefede5b58ba3d1f [term 1 FOLLOWER]: Leader election won for term 1
I20260923 20:12:18.557861 15569 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2d0baa738a1b4dbcaefede5b58ba3d1f [term 1 LEADER]: Becoming Leader. State: Replica: 2d0baa738a1b4dbcaefede5b58ba3d1f, State: Running, Role: LEADER
I20260923 20:12:18.558418 15569 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2d0baa738a1b4dbcaefede5b58ba3d1f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d0baa738a1b4dbcaefede5b58ba3d1f" member_type: VOTER last_known_addr { host: "127.1.170.190" port: 43771 } }
I20260923 20:12:18.558995 15564 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2d0baa738a1b4dbcaefede5b58ba3d1f [sys.catalog]: configured and running, proceeding with master startup.
I20260923 20:12:18.561471 15570 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2d0baa738a1b4dbcaefede5b58ba3d1f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2d0baa738a1b4dbcaefede5b58ba3d1f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d0baa738a1b4dbcaefede5b58ba3d1f" member_type: VOTER last_known_addr { host: "127.1.170.190" port: 43771 } } }
I20260923 20:12:18.561384 15571 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2d0baa738a1b4dbcaefede5b58ba3d1f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2d0baa738a1b4dbcaefede5b58ba3d1f. Latest consensus state: current_term: 1 leader_uuid: "2d0baa738a1b4dbcaefede5b58ba3d1f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d0baa738a1b4dbcaefede5b58ba3d1f" member_type: VOTER last_known_addr { host: "127.1.170.190" port: 43771 } } }
I20260923 20:12:18.562072 15570 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2d0baa738a1b4dbcaefede5b58ba3d1f [sys.catalog]: This master's current role is: LEADER
I20260923 20:12:18.562095 15571 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2d0baa738a1b4dbcaefede5b58ba3d1f [sys.catalog]: This master's current role is: LEADER
I20260923 20:12:18.564384 15578 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 20:12:18.568676 15578 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 20:12:18.577052 15578 catalog_manager.cc:1347] Generated new cluster ID: f99fda99d9d34b7e9dbf93ba0affc942
I20260923 20:12:18.577188 15578 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 20:12:18.595273 15578 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 20:12:18.596679 15578 catalog_manager.cc:1504] Loading token signing keys...
I20260923 20:12:18.606001 15578 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2d0baa738a1b4dbcaefede5b58ba3d1f: Generated new TSK 0
I20260923 20:12:18.606889 15578 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 20:12:18.661636 1706 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskqUuku6/build/asan/bin/kudu pid 15497
I20260923 20:12:18.663528 15559 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:12:18.763083 1706 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqUuku6/build/asan/bin/kudu with pid 15497
2026-09-23T20:12:18Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (2100 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194121577027-1706-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 20:12:18 dist-test-slave-th4g krb5kdc[15592](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 20:12:18 dist-test-slave-th4g krb5kdc[15592](info): set up 2 sockets
Sep 23 20:12:18 dist-test-slave-th4g krb5kdc[15592](info): commencing operation
krb5kdc: starting...
W20260923 20:12:19.898488 1706 mini_kdc.cc:121] Time spent starting KDC: real 1.097s user 0.013s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194121577027-1706-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194121577027-1706-0/krb5kdc/test-user.keytab.
Sep 23 20:12:20 dist-test-slave-th4g krb5kdc[15592](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194340, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T20:12:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T20:12:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.170.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.170.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.170.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194121577027-1706-0/krb5kdc/kudu_127.1.170.190.keytab.
Entry for principal kudu/127.1.170.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194121577027-1706-0/krb5kdc/kudu_127.1.170.190.keytab.
I20260923 20:12:20.138367 1706 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqUuku6/build/asan/bin/kudu
/tmp/dist-test-taskqUuku6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194121577027-1706-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194121577027-1706-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194121577027-1706-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194121577027-1706-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.170.190:43769
--webserver_interface=127.1.170.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194121577027-1706-0/krb5kdc/kudu_127.1.170.190.keytab
--principal=kudu/127.1.170.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.170.148:37943
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.170.190:43769
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194121577027-1706-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194121577027-1706-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194121577027-1706-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:12:20.469156 15605 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:12:20.469440 15605 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:12:20.469529 15605 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:12:20.476702 15605 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 20:12:20.476809 15605 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:12:20.476855 15605 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:12:20.476912 15605 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 20:12:20.476955 15605 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 20:12:20.486680 15605 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.170.148:37943
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194121577027-1706-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194121577027-1706-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.170.190:43769
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194121577027-1706-0/krb5kdc/kudu_127.1.170.190.keytab
--principal=kudu/127.1.170.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.170.190:43769
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194121577027-1706-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.170.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194121577027-1706-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:12:20.487854 15605 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:12:20.489111 15605 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:12:20.500715 15610 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:12:20.500787 15611 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:12:20.500813 15613 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:12:20.501830 15605 server_base.cc:1034] running on GCE node
I20260923 20:12:20.502673 15605 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:12:20.504081 15605 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:12:20.505280 15605 hybrid_clock.cc:648] HybridClock initialized: now 1790194340505226 us; error 49 us; skew 500 ppm
I20260923 20:12:20.505620 15605 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:12:20 dist-test-slave-th4g krb5kdc[15592](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194340, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.170.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:12:20.508786 15605 init.cc:377] Logged in from keytab as kudu/127.1.170.190@KRBTEST.COM (short username kudu)
I20260923 20:12:20.510265 15605 webserver.cc:460] Webserver started at http://127.1.170.190:39057/ using document root <none> and password file <none>
I20260923 20:12:20.510895 15605 fs_manager.cc:359] Metadata directory not provided
I20260923 20:12:20.510993 15605 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:12:20.511242 15605 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:12:20.514209 15605 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194121577027-1706-0/minicluster-data/master-0/data/instance:
uuid: "e910e26b0d9645f0bca3a3198101aa5a"
format_stamp: "Formatted at 2026-09-23 20:12:20 on dist-test-slave-th4g"
server_key: "0322a97aa01b78f36a69b9b99ac35141"
server_key_iv: "469c0c9a2ee402c69b62f6ea74f31bac"
server_key_version: "encryptionkey@0"
I20260923 20:12:20.514925 15605 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194121577027-1706-0/minicluster-data/master-0/wal/instance:
uuid: "e910e26b0d9645f0bca3a3198101aa5a"
format_stamp: "Formatted at 2026-09-23 20:12:20 on dist-test-slave-th4g"
server_key: "0322a97aa01b78f36a69b9b99ac35141"
server_key_iv: "469c0c9a2ee402c69b62f6ea74f31bac"
server_key_version: "encryptionkey@0"
I20260923 20:12:20.520027 15605 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260923 20:12:20.523475 15620 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:12:20.524791 15605 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260923 20:12:20.524991 15605 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194121577027-1706-0/minicluster-data/master-0/data,/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194121577027-1706-0/minicluster-data/master-0/wal
uuid: "e910e26b0d9645f0bca3a3198101aa5a"
format_stamp: "Formatted at 2026-09-23 20:12:20 on dist-test-slave-th4g"
server_key: "0322a97aa01b78f36a69b9b99ac35141"
server_key_iv: "469c0c9a2ee402c69b62f6ea74f31bac"
server_key_version: "encryptionkey@0"
I20260923 20:12:20.525198 15605 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194121577027-1706-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194121577027-1706-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194121577027-1706-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:12:20.540505 15605 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:12:20.544449 15605 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:12:20.544742 15605 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:12:20.563134 15605 rpc_server.cc:307] RPC server started. Bound to: 127.1.170.190:43769
I20260923 20:12:20.563181 15671 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.170.190:43769 every 8 connection(s)
I20260923 20:12:20.564477 15605 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194121577027-1706-0/minicluster-data/master-0/data/info.pb
I20260923 20:12:20.568050 15672 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:12:20.569623 1706 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqUuku6/build/asan/bin/kudu as pid 15605
I20260923 20:12:20.569828 1706 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194121577027-1706-0/minicluster-data/master-0/wal/instance
I20260923 20:12:20.570490 1706 external_mini_cluster.cc:1428] Setting key 290883508a3152d940439393b0e97b6b
I20260923 20:12:20.581291 15672 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e910e26b0d9645f0bca3a3198101aa5a: Bootstrap starting.
I20260923 20:12:20.584847 15672 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e910e26b0d9645f0bca3a3198101aa5a: Neither blocks nor log segments found. Creating new log.
Sep 23 20:12:20 dist-test-slave-th4g krb5kdc[15592](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194340, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.170.190@KRBTEST.COM
I20260923 20:12:20.585934 15672 log.cc:824] T 00000000000000000000000000000000 P e910e26b0d9645f0bca3a3198101aa5a: Log is configured to *not* fsync() on all Append() calls
I20260923 20:12:20.589332 15672 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e910e26b0d9645f0bca3a3198101aa5a: No bootstrap required, opened a new log
I20260923 20:12:20.595295 15675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:12:20.572480 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55484 (local address 127.1.170.190:43769)
0923 20:12:20.573385 (+ 905us) server_negotiation.cc:206] Beginning negotiation
0923 20:12:20.573400 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0923 20:12:20.573455 (+ 55us) server_negotiation.cc:407] Connection header received
0923 20:12:20.575797 (+ 2342us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:12:20.575825 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:12:20.576320 (+ 495us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:12:20.576750 (+ 430us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:12:20.578200 (+ 1450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:12:20.580273 (+ 2073us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:12:20.581661 (+ 1388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:12:20.582642 (+ 981us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:12:20.582818 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:12:20.586865 (+ 4047us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:12:20.586873 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:12:20.586886 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:12:20.586919 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:12:20.590784 (+ 3865us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:12:20.591462 (+ 678us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:12:20.591471 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:12:20.591479 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:12:20.591691 (+ 212us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:12:20.592041 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:12:20.592046 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:12:20.592050 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:12:20.592764 (+ 714us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:12:20.593106 (+ 342us) server_negotiation.cc:1035] Waiting for connection context
0923 20:12:20.593500 (+ 394us) server_negotiation.cc:299] Negotiation successful
0923 20:12:20.593972 (+ 472us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":628,"thread_start_us":376,"threads_started":1}
I20260923 20:12:20.599375 15672 raft_consensus.cc:348] T 00000000000000000000000000000000 P e910e26b0d9645f0bca3a3198101aa5a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e910e26b0d9645f0bca3a3198101aa5a" member_type: VOTER last_known_addr { host: "127.1.170.190" port: 43769 } }
I20260923 20:12:20.599709 15672 raft_consensus.cc:374] T 00000000000000000000000000000000 P e910e26b0d9645f0bca3a3198101aa5a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:12:20.599781 15672 raft_consensus.cc:729] T 00000000000000000000000000000000 P e910e26b0d9645f0bca3a3198101aa5a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e910e26b0d9645f0bca3a3198101aa5a, State: Initialized, Role: FOLLOWER
I20260923 20:12:20.600217 15672 consensus_queue.cc:260] T 00000000000000000000000000000000 P e910e26b0d9645f0bca3a3198101aa5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e910e26b0d9645f0bca3a3198101aa5a" member_type: VOTER last_known_addr { host: "127.1.170.190" port: 43769 } }
I20260923 20:12:20.600401 15672 raft_consensus.cc:388] T 00000000000000000000000000000000 P e910e26b0d9645f0bca3a3198101aa5a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 20:12:20.600495 15672 raft_consensus.cc:482] T 00000000000000000000000000000000 P e910e26b0d9645f0bca3a3198101aa5a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 20:12:20.600620 15672 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e910e26b0d9645f0bca3a3198101aa5a [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:12:20.602735 15672 raft_consensus.cc:504] T 00000000000000000000000000000000 P e910e26b0d9645f0bca3a3198101aa5a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e910e26b0d9645f0bca3a3198101aa5a" member_type: VOTER last_known_addr { host: "127.1.170.190" port: 43769 } }
I20260923 20:12:20.603159 15672 leader_election.cc:302] T 00000000000000000000000000000000 P e910e26b0d9645f0bca3a3198101aa5a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e910e26b0d9645f0bca3a3198101aa5a; no voters:
I20260923 20:12:20.603735 15672 leader_election.cc:288] T 00000000000000000000000000000000 P e910e26b0d9645f0bca3a3198101aa5a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 20:12:20.603849 15677 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e910e26b0d9645f0bca3a3198101aa5a [term 1 FOLLOWER]: Leader election won for term 1
I20260923 20:12:20.604919 15677 raft_consensus.cc:686] T 00000000000000000000000000000000 P e910e26b0d9645f0bca3a3198101aa5a [term 1 LEADER]: Becoming Leader. State: Replica: e910e26b0d9645f0bca3a3198101aa5a, State: Running, Role: LEADER
I20260923 20:12:20.605398 15677 consensus_queue.cc:237] T 00000000000000000000000000000000 P e910e26b0d9645f0bca3a3198101aa5a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e910e26b0d9645f0bca3a3198101aa5a" member_type: VOTER last_known_addr { host: "127.1.170.190" port: 43769 } }
I20260923 20:12:20.605851 15672 sys_catalog.cc:564] T 00000000000000000000000000000000 P e910e26b0d9645f0bca3a3198101aa5a [sys.catalog]: configured and running, proceeding with master startup.
I20260923 20:12:20.608428 15679 sys_catalog.cc:455] T 00000000000000000000000000000000 P e910e26b0d9645f0bca3a3198101aa5a [sys.catalog]: SysCatalogTable state changed. Reason: New leader e910e26b0d9645f0bca3a3198101aa5a. Latest consensus state: current_term: 1 leader_uuid: "e910e26b0d9645f0bca3a3198101aa5a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e910e26b0d9645f0bca3a3198101aa5a" member_type: VOTER last_known_addr { host: "127.1.170.190" port: 43769 } } }
I20260923 20:12:20.609086 15679 sys_catalog.cc:458] T 00000000000000000000000000000000 P e910e26b0d9645f0bca3a3198101aa5a [sys.catalog]: This master's current role is: LEADER
I20260923 20:12:20.608819 15678 sys_catalog.cc:455] T 00000000000000000000000000000000 P e910e26b0d9645f0bca3a3198101aa5a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e910e26b0d9645f0bca3a3198101aa5a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e910e26b0d9645f0bca3a3198101aa5a" member_type: VOTER last_known_addr { host: "127.1.170.190" port: 43769 } } }
I20260923 20:12:20.610080 15678 sys_catalog.cc:458] T 00000000000000000000000000000000 P e910e26b0d9645f0bca3a3198101aa5a [sys.catalog]: This master's current role is: LEADER
I20260923 20:12:20.611215 15683 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 20:12:20.616302 15683 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 20:12:20.625526 15683 catalog_manager.cc:1347] Generated new cluster ID: 26a0561cd2ff469cb55904b448b78043
I20260923 20:12:20.625655 15683 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 20:12:20.641022 15683 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 20:12:20.642338 15683 catalog_manager.cc:1504] Loading token signing keys...
I20260923 20:12:20.650208 15683 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e910e26b0d9645f0bca3a3198101aa5a: Generated new TSK 0
I20260923 20:12:20.650969 15683 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.170.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.170.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.170.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194121577027-1706-0/krb5kdc/kudu_127.1.170.129.keytab.
Entry for principal kudu/127.1.170.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194121577027-1706-0/krb5kdc/kudu_127.1.170.129.keytab.
I20260923 20:12:20.721481 1706 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqUuku6/build/asan/bin/kudu
/tmp/dist-test-taskqUuku6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194121577027-1706-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194121577027-1706-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194121577027-1706-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194121577027-1706-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.170.129:0
--local_ip_for_outbound_sockets=127.1.170.129
--webserver_interface=127.1.170.129
--webserver_port=0
--tserver_master_addrs=127.1.170.190:43769
--keytab_file=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194121577027-1706-0/krb5kdc/kudu_127.1.170.129.keytab
--principal=kudu/127.1.170.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.170.148:37943
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194121577027-1706-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194121577027-1706-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194121577027-1706-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:12:21.058982 15698 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:12:21.059264 15698 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:12:21.059360 15698 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:12:21.066543 15698 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:12:21.066648 15698 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:12:21.066761 15698 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.170.129
I20260923 20:12:21.076098 15698 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.170.148:37943
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194121577027-1706-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194121577027-1706-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194121577027-1706-0/krb5kdc/kudu_127.1.170.129.keytab
--principal=kudu/127.1.170.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.170.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194121577027-1706-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.170.129
--webserver_port=0
--tserver_master_addrs=127.1.170.190:43769
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.170.129
--log_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194121577027-1706-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:12:21.077273 15698 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:12:21.078591 15698 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:12:21.092304 15704 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:12:21.092341 15706 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:12:21.092370 15703 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:12:21.093605 15698 server_base.cc:1034] running on GCE node
I20260923 20:12:21.094301 15698 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:12:21.095310 15698 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:12:21.096621 15698 hybrid_clock.cc:648] HybridClock initialized: now 1790194341096567 us; error 33 us; skew 500 ppm
I20260923 20:12:21.097025 15698 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:12:21 dist-test-slave-th4g krb5kdc[15592](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194341, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.170.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:12:21.100373 15698 init.cc:377] Logged in from keytab as kudu/127.1.170.129@KRBTEST.COM (short username kudu)
I20260923 20:12:21.101773 15698 webserver.cc:460] Webserver started at http://127.1.170.129:42049/ using document root <none> and password file <none>
I20260923 20:12:21.102484 15698 fs_manager.cc:359] Metadata directory not provided
I20260923 20:12:21.102638 15698 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:12:21.102933 15698 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:12:21.105926 15698 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194121577027-1706-0/minicluster-data/ts-0/data/instance:
uuid: "7f5b750bad6042cda73694b87b237180"
format_stamp: "Formatted at 2026-09-23 20:12:21 on dist-test-slave-th4g"
server_key: "0ef88f7572d94c5b107314394af66393"
server_key_iv: "4fc80b00dd9f302d00610d10530c0ad8"
server_key_version: "encryptionkey@0"
I20260923 20:12:21.106731 15698 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194121577027-1706-0/minicluster-data/ts-0/wal/instance:
uuid: "7f5b750bad6042cda73694b87b237180"
format_stamp: "Formatted at 2026-09-23 20:12:21 on dist-test-slave-th4g"
server_key: "0ef88f7572d94c5b107314394af66393"
server_key_iv: "4fc80b00dd9f302d00610d10530c0ad8"
server_key_version: "encryptionkey@0"
I20260923 20:12:21.112071 15698 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260923 20:12:21.115891 15713 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:12:21.117328 15698 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260923 20:12:21.117617 15698 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194121577027-1706-0/minicluster-data/ts-0/data,/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194121577027-1706-0/minicluster-data/ts-0/wal
uuid: "7f5b750bad6042cda73694b87b237180"
format_stamp: "Formatted at 2026-09-23 20:12:21 on dist-test-slave-th4g"
server_key: "0ef88f7572d94c5b107314394af66393"
server_key_iv: "4fc80b00dd9f302d00610d10530c0ad8"
server_key_version: "encryptionkey@0"
I20260923 20:12:21.117834 15698 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194121577027-1706-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194121577027-1706-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194121577027-1706-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:12:21.137229 15698 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:12:21.141353 15698 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:12:21.141589 15698 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:12:21.142562 15698 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 20:12:21.143815 15698 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 20:12:21.143888 15698 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 20:12:21.143972 15698 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 20:12:21.144016 15698 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 20:12:21.177541 15698 rpc_server.cc:307] RPC server started. Bound to: 127.1.170.129:38865
I20260923 20:12:21.177583 15825 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.170.129:38865 every 8 connection(s)
I20260923 20:12:21.178840 15698 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194121577027-1706-0/minicluster-data/ts-0/data/info.pb
I20260923 20:12:21.179385 1706 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqUuku6/build/asan/bin/kudu as pid 15698
I20260923 20:12:21.179618 1706 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194121577027-1706-0/minicluster-data/ts-0/wal/instance
I20260923 20:12:21.180440 1706 external_mini_cluster.cc:1428] Setting key 24d2a55f58f366713a593e1360dc49b9
Sep 23 20:12:21 dist-test-slave-th4g krb5kdc[15592](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194341, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.170.129@KRBTEST.COM for kudu/127.1.170.190@KRBTEST.COM
I20260923 20:12:21.200369 15828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:12:21.181439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.170.129:55837 (local address 127.1.170.190:43769)
0923 20:12:21.181967 (+ 528us) server_negotiation.cc:206] Beginning negotiation
0923 20:12:21.181971 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 20:12:21.182985 (+ 1014us) server_negotiation.cc:407] Connection header received
0923 20:12:21.184743 (+ 1758us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:12:21.184748 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:12:21.184830 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:12:21.184952 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:12:21.186487 (+ 1535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:12:21.187209 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:12:21.188852 (+ 1643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:12:21.189238 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:12:21.189344 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:12:21.193168 (+ 3824us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:12:21.193177 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:12:21.193185 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:12:21.193232 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:12:21.196906 (+ 3674us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:12:21.197658 (+ 752us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:12:21.197665 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:12:21.197672 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:12:21.197822 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:12:21.198282 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:12:21.198290 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:12:21.198296 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:12:21.198722 (+ 426us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:12:21.198881 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0923 20:12:21.199581 (+ 700us) server_negotiation.cc:299] Negotiation successful
0923 20:12:21.199891 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":375,"thread_start_us":275,"threads_started":1}
I20260923 20:12:21.201081 15829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:12:21.182090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.190:43769 (local address 127.1.170.129:55837)
0923 20:12:21.182842 (+ 752us) negotiation.cc:107] Waiting for socket to connect
0923 20:12:21.182878 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0923 20:12:21.184485 (+ 1607us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:12:21.185185 (+ 700us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:12:21.185193 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:12:21.185681 (+ 488us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:12:21.186315 (+ 634us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:12:21.186332 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:12:21.187383 (+ 1051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:12:21.187388 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:12:21.188681 (+ 1293us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:12:21.188698 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:12:21.189483 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:12:21.189492 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:12:21.189796 (+ 304us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:12:21.190523 (+ 727us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:12:21.190550 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:12:21.192940 (+ 2390us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:12:21.197057 (+ 4117us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:12:21.197064 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:12:21.197081 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:12:21.197502 (+ 421us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:12:21.197963 (+ 461us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:12:21.197966 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:12:21.197969 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:12:21.198153 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:12:21.198852 (+ 699us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:12:21.198859 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:12:21.199188 (+ 329us) client_negotiation.cc:769] Sending connection context
0923 20:12:21.199423 (+ 235us) client_negotiation.cc:240] Negotiation successful
0923 20:12:21.199874 (+ 451us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":523,"thread_start_us":289,"threads_started":1}
I20260923 20:12:21.202694 15826 heartbeater.cc:347] Connected to a master server at 127.1.170.190:43769
I20260923 20:12:21.202983 15826 heartbeater.cc:464] Registering TS with master...
I20260923 20:12:21.203697 15826 heartbeater.cc:511] Master 127.1.170.190:43769 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.1.170.130@KRBTEST.COM; defaulting to no policy
I20260923 20:12:21.205842 15637 ts_manager.cc:194] Registered new tserver with Master: 7f5b750bad6042cda73694b87b237180 (127.1.170.129:38865)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.170.130@KRBTEST.COM" created.
I20260923 20:12:21.207891 15637 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.170.129@KRBTEST.COM'} at 127.1.170.129:55837
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.170.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194121577027-1706-0/krb5kdc/kudu_127.1.170.130.keytab.
Entry for principal kudu/127.1.170.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194121577027-1706-0/krb5kdc/kudu_127.1.170.130.keytab.
I20260923 20:12:21.235347 1706 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqUuku6/build/asan/bin/kudu
/tmp/dist-test-taskqUuku6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194121577027-1706-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194121577027-1706-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194121577027-1706-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194121577027-1706-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.170.130:0
--local_ip_for_outbound_sockets=127.1.170.130
--webserver_interface=127.1.170.130
--webserver_port=0
--tserver_master_addrs=127.1.170.190:43769
--keytab_file=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194121577027-1706-0/krb5kdc/kudu_127.1.170.130.keytab
--principal=kudu/127.1.170.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.170.148:37943
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194121577027-1706-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194121577027-1706-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194121577027-1706-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:12:21.568027 15832 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:12:21.568356 15832 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:12:21.568526 15832 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:12:21.575989 15832 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:12:21.576162 15832 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:12:21.576352 15832 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.170.130
I20260923 20:12:21.586231 15832 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.170.148:37943
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194121577027-1706-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194121577027-1706-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194121577027-1706-0/krb5kdc/kudu_127.1.170.130.keytab
--principal=kudu/127.1.170.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.170.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194121577027-1706-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.170.130
--webserver_port=0
--tserver_master_addrs=127.1.170.190:43769
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.170.130
--log_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194121577027-1706-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:12:21.587627 15832 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:12:21.588990 15832 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:12:21.602372 15837 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:12:21.602433 15838 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:12:21.602433 15840 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:12:21.603711 15832 server_base.cc:1034] running on GCE node
I20260923 20:12:21.604370 15832 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:12:21.605377 15832 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:12:21.606680 15832 hybrid_clock.cc:648] HybridClock initialized: now 1790194341606564 us; error 59 us; skew 500 ppm
I20260923 20:12:21.607052 15832 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:12:21 dist-test-slave-th4g krb5kdc[15592](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194341, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.170.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:12:21.610335 15832 init.cc:377] Logged in from keytab as kudu/127.1.170.130@KRBTEST.COM (short username kudu)
I20260923 20:12:21.611923 15832 webserver.cc:460] Webserver started at http://127.1.170.130:41715/ using document root <none> and password file <none>
I20260923 20:12:21.612617 15832 fs_manager.cc:359] Metadata directory not provided
I20260923 20:12:21.612738 15832 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:12:21.613031 15832 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:12:21.616333 15832 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194121577027-1706-0/minicluster-data/ts-1/data/instance:
uuid: "53297953fdab4397adf94a117c453bc4"
format_stamp: "Formatted at 2026-09-23 20:12:21 on dist-test-slave-th4g"
server_key: "3f95013455082ede0c61607aa4f14e3e"
server_key_iv: "546d26b1ce46b518a3cf9e42439abc2f"
server_key_version: "encryptionkey@0"
I20260923 20:12:21.617264 15832 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194121577027-1706-0/minicluster-data/ts-1/wal/instance:
uuid: "53297953fdab4397adf94a117c453bc4"
format_stamp: "Formatted at 2026-09-23 20:12:21 on dist-test-slave-th4g"
server_key: "3f95013455082ede0c61607aa4f14e3e"
server_key_iv: "546d26b1ce46b518a3cf9e42439abc2f"
server_key_version: "encryptionkey@0"
I20260923 20:12:21.623046 15832 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.005s
I20260923 20:12:21.626617 15847 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:12:21.628228 15832 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260923 20:12:21.628368 15832 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194121577027-1706-0/minicluster-data/ts-1/data,/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194121577027-1706-0/minicluster-data/ts-1/wal
uuid: "53297953fdab4397adf94a117c453bc4"
format_stamp: "Formatted at 2026-09-23 20:12:21 on dist-test-slave-th4g"
server_key: "3f95013455082ede0c61607aa4f14e3e"
server_key_iv: "546d26b1ce46b518a3cf9e42439abc2f"
server_key_version: "encryptionkey@0"
I20260923 20:12:21.628554 15832 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194121577027-1706-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194121577027-1706-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194121577027-1706-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:12:21.639959 15832 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:12:21.644424 15832 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:12:21.644744 15832 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:12:21.645777 15832 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 20:12:21.647225 15832 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 20:12:21.647306 15832 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 20:12:21.647425 15832 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 20:12:21.647483 15832 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 20:12:21.682940 15832 rpc_server.cc:307] RPC server started. Bound to: 127.1.170.130:37359
I20260923 20:12:21.682976 15959 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.170.130:37359 every 8 connection(s)
I20260923 20:12:21.684229 15832 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194121577027-1706-0/minicluster-data/ts-1/data/info.pb
I20260923 20:12:21.688519 1706 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqUuku6/build/asan/bin/kudu as pid 15832
I20260923 20:12:21.688705 1706 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194121577027-1706-0/minicluster-data/ts-1/wal/instance
I20260923 20:12:21.689162 1706 external_mini_cluster.cc:1428] Setting key 15bf2b1e7f2204f4264b4a508edb6414
Sep 23 20:12:21 dist-test-slave-th4g krb5kdc[15592](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194341, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.170.130@KRBTEST.COM for kudu/127.1.170.190@KRBTEST.COM
I20260923 20:12:21.706629 15828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:12:21.686901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.170.130:51337 (local address 127.1.170.190:43769)
0923 20:12:21.687106 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0923 20:12:21.687111 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 20:12:21.688252 (+ 1141us) server_negotiation.cc:407] Connection header received
0923 20:12:21.690614 (+ 2362us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:12:21.690619 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:12:21.690762 (+ 143us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:12:21.690872 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:12:21.692508 (+ 1636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:12:21.693204 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:12:21.694797 (+ 1593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:12:21.695216 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:12:21.695349 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:12:21.698837 (+ 3488us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:12:21.698842 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:12:21.698845 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:12:21.698878 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:12:21.702506 (+ 3628us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:12:21.703397 (+ 891us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:12:21.703407 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:12:21.703414 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:12:21.703679 (+ 265us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:12:21.704277 (+ 598us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:12:21.704286 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:12:21.704292 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:12:21.704690 (+ 398us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:12:21.704869 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0923 20:12:21.705720 (+ 851us) server_negotiation.cc:299] Negotiation successful
0923 20:12:21.706082 (+ 362us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260923 20:12:21.707397 15962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:12:21.687336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.190:43769 (local address 127.1.170.130:51337)
0923 20:12:21.688061 (+ 725us) negotiation.cc:107] Waiting for socket to connect
0923 20:12:21.688116 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0923 20:12:21.690277 (+ 2161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:12:21.691184 (+ 907us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:12:21.691195 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:12:21.691781 (+ 586us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:12:21.692338 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:12:21.692355 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:12:21.693358 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:12:21.693364 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:12:21.694650 (+ 1286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:12:21.694662 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:12:21.695440 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:12:21.695447 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:12:21.695671 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:12:21.696392 (+ 721us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:12:21.696414 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:12:21.698606 (+ 2192us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:12:21.702664 (+ 4058us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:12:21.702672 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:12:21.702687 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:12:21.703188 (+ 501us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:12:21.703829 (+ 641us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:12:21.703842 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:12:21.703844 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:12:21.704140 (+ 296us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:12:21.704847 (+ 707us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:12:21.704854 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:12:21.705247 (+ 393us) client_negotiation.cc:769] Sending connection context
0923 20:12:21.705522 (+ 275us) client_negotiation.cc:240] Negotiation successful
0923 20:12:21.706130 (+ 608us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":512,"thread_start_us":312,"threads_started":1}
I20260923 20:12:21.709297 15960 heartbeater.cc:347] Connected to a master server at 127.1.170.190:43769
I20260923 20:12:21.709652 15960 heartbeater.cc:464] Registering TS with master...
I20260923 20:12:21.710637 15960 heartbeater.cc:511] Master 127.1.170.190:43769 requested a full tablet report, sending...
I20260923 20:12:21.712669 15637 ts_manager.cc:194] Registered new tserver with Master: 53297953fdab4397adf94a117c453bc4 (127.1.170.130:37359)
I20260923 20:12:21.713572 15637 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.170.130@KRBTEST.COM'} at 127.1.170.130:51337
WARNING: no policy specified for kudu/127.1.170.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.170.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.170.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194121577027-1706-0/krb5kdc/kudu_127.1.170.131.keytab.
Entry for principal kudu/127.1.170.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194121577027-1706-0/krb5kdc/kudu_127.1.170.131.keytab.
I20260923 20:12:21.747500 1706 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqUuku6/build/asan/bin/kudu
/tmp/dist-test-taskqUuku6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194121577027-1706-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194121577027-1706-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194121577027-1706-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194121577027-1706-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.170.131:0
--local_ip_for_outbound_sockets=127.1.170.131
--webserver_interface=127.1.170.131
--webserver_port=0
--tserver_master_addrs=127.1.170.190:43769
--keytab_file=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194121577027-1706-0/krb5kdc/kudu_127.1.170.131.keytab
--principal=kudu/127.1.170.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.170.148:37943
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194121577027-1706-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194121577027-1706-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194121577027-1706-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:12:22.084219 15965 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:12:22.084551 15965 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:12:22.084735 15965 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:12:22.092219 15965 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:12:22.092386 15965 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:12:22.092566 15965 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.170.131
I20260923 20:12:22.102881 15965 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.170.148:37943
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194121577027-1706-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194121577027-1706-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194121577027-1706-0/krb5kdc/kudu_127.1.170.131.keytab
--principal=kudu/127.1.170.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.170.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194121577027-1706-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.170.131
--webserver_port=0
--tserver_master_addrs=127.1.170.190:43769
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.170.131
--log_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194121577027-1706-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:12:22.104290 15965 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:12:22.105584 15965 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:12:22.119266 15970 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:12:22.119211 15973 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:12:22.119249 15971 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:12:22.120493 15965 server_base.cc:1034] running on GCE node
I20260923 20:12:22.121188 15965 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:12:22.122304 15965 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:12:22.123546 15965 hybrid_clock.cc:648] HybridClock initialized: now 1790194342123500 us; error 40 us; skew 500 ppm
I20260923 20:12:22.123893 15965 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:12:22 dist-test-slave-th4g krb5kdc[15592](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194342, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.170.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:12:22.127602 15965 init.cc:377] Logged in from keytab as kudu/127.1.170.131@KRBTEST.COM (short username kudu)
I20260923 20:12:22.129305 15965 webserver.cc:460] Webserver started at http://127.1.170.131:34681/ using document root <none> and password file <none>
I20260923 20:12:22.130138 15965 fs_manager.cc:359] Metadata directory not provided
I20260923 20:12:22.130250 15965 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:12:22.130522 15965 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:12:22.134090 15965 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194121577027-1706-0/minicluster-data/ts-2/data/instance:
uuid: "d06de688bc57487a9a34608cd7c70b6a"
format_stamp: "Formatted at 2026-09-23 20:12:22 on dist-test-slave-th4g"
server_key: "6f84fe378bd8c0618a4fe5ebbe74fe2e"
server_key_iv: "344643454d475f00f5172ce964fc04f8"
server_key_version: "encryptionkey@0"
I20260923 20:12:22.134868 15965 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194121577027-1706-0/minicluster-data/ts-2/wal/instance:
uuid: "d06de688bc57487a9a34608cd7c70b6a"
format_stamp: "Formatted at 2026-09-23 20:12:22 on dist-test-slave-th4g"
server_key: "6f84fe378bd8c0618a4fe5ebbe74fe2e"
server_key_iv: "344643454d475f00f5172ce964fc04f8"
server_key_version: "encryptionkey@0"
I20260923 20:12:22.140443 15965 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260923 20:12:22.144227 15980 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:12:22.145720 15965 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260923 20:12:22.145931 15965 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194121577027-1706-0/minicluster-data/ts-2/data,/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194121577027-1706-0/minicluster-data/ts-2/wal
uuid: "d06de688bc57487a9a34608cd7c70b6a"
format_stamp: "Formatted at 2026-09-23 20:12:22 on dist-test-slave-th4g"
server_key: "6f84fe378bd8c0618a4fe5ebbe74fe2e"
server_key_iv: "344643454d475f00f5172ce964fc04f8"
server_key_version: "encryptionkey@0"
I20260923 20:12:22.146132 15965 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194121577027-1706-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194121577027-1706-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194121577027-1706-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:12:22.182379 15965 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:12:22.187366 15965 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:12:22.187624 15965 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:12:22.188524 15965 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 20:12:22.190137 15965 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 20:12:22.190225 15965 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 20:12:22.190323 15965 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 20:12:22.190368 15965 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 20:12:22.211943 15826 heartbeater.cc:503] Master 127.1.170.190:43769 was elected leader, sending a full tablet report...
I20260923 20:12:22.226854 15965 rpc_server.cc:307] RPC server started. Bound to: 127.1.170.131:40407
I20260923 20:12:22.226898 16092 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.170.131:40407 every 8 connection(s)
I20260923 20:12:22.228205 15965 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194121577027-1706-0/minicluster-data/ts-2/data/info.pb
I20260923 20:12:22.235060 1706 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqUuku6/build/asan/bin/kudu as pid 15965
I20260923 20:12:22.235262 1706 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194121577027-1706-0/minicluster-data/ts-2/wal/instance
I20260923 20:12:22.235689 1706 external_mini_cluster.cc:1428] Setting key 45aed41da1f2ea4ba065cfc1945ed404
Sep 23 20:12:22 dist-test-slave-th4g krb5kdc[15592](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194342, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.170.131@KRBTEST.COM for kudu/127.1.170.190@KRBTEST.COM
I20260923 20:12:22.250232 16095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:12:22.230834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.170.131:36799 (local address 127.1.170.190:43769)
0923 20:12:22.231298 (+ 464us) server_negotiation.cc:206] Beginning negotiation
0923 20:12:22.231303 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 20:12:22.232180 (+ 877us) server_negotiation.cc:407] Connection header received
0923 20:12:22.234085 (+ 1905us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:12:22.234091 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:12:22.234188 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:12:22.234307 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:12:22.235921 (+ 1614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:12:22.236907 (+ 986us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:12:22.238899 (+ 1992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:12:22.239521 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:12:22.239627 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:12:22.243987 (+ 4360us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:12:22.243993 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:12:22.243996 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:12:22.244034 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:12:22.246853 (+ 2819us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:12:22.247651 (+ 798us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:12:22.247655 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:12:22.247657 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:12:22.247781 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:12:22.248242 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:12:22.248245 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:12:22.248247 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:12:22.248514 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:12:22.248623 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0923 20:12:22.249515 (+ 892us) server_negotiation.cc:299] Negotiation successful
0923 20:12:22.249758 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":356,"thread_start_us":260,"threads_started":1}
I20260923 20:12:22.251261 16096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:12:22.231290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.190:43769 (local address 127.1.170.131:36799)
0923 20:12:22.231996 (+ 706us) negotiation.cc:107] Waiting for socket to connect
0923 20:12:22.232032 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0923 20:12:22.233772 (+ 1740us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:12:22.234539 (+ 767us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:12:22.234546 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:12:22.235047 (+ 501us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:12:22.235699 (+ 652us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:12:22.235716 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:12:22.237135 (+ 1419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:12:22.237140 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:12:22.238729 (+ 1589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:12:22.238743 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:12:22.239772 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:12:22.239778 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:12:22.239999 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:12:22.240661 (+ 662us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:12:22.240694 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:12:22.243708 (+ 3014us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:12:22.247046 (+ 3338us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:12:22.247053 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:12:22.247068 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:12:22.247519 (+ 451us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:12:22.247901 (+ 382us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:12:22.247905 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:12:22.247907 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:12:22.248138 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:12:22.248647 (+ 509us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:12:22.248654 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:12:22.249003 (+ 349us) client_negotiation.cc:769] Sending connection context
0923 20:12:22.249317 (+ 314us) client_negotiation.cc:240] Negotiation successful
0923 20:12:22.249828 (+ 511us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":468,"thread_start_us":302,"threads_started":1}
I20260923 20:12:22.252527 16093 heartbeater.cc:347] Connected to a master server at 127.1.170.190:43769
I20260923 20:12:22.252794 16093 heartbeater.cc:464] Registering TS with master...
I20260923 20:12:22.253460 16093 heartbeater.cc:511] Master 127.1.170.190:43769 requested a full tablet report, sending...
I20260923 20:12:22.255291 15637 ts_manager.cc:194] Registered new tserver with Master: d06de688bc57487a9a34608cd7c70b6a (127.1.170.131:40407)
I20260923 20:12:22.256247 15637 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.170.131@KRBTEST.COM'} at 127.1.170.131:36799
I20260923 20:12:22.264734 1706 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260923 20:12:22.282342 16095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:12:22.269233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55492 (local address 127.1.170.190:43769)
0923 20:12:22.269420 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0923 20:12:22.269425 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 20:12:22.269807 (+ 382us) server_negotiation.cc:407] Connection header received
0923 20:12:22.270026 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:12:22.270029 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:12:22.270109 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:12:22.270199 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:12:22.271546 (+ 1347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:12:22.272279 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:12:22.273692 (+ 1413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:12:22.274197 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:12:22.274295 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:12:22.276813 (+ 2518us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:12:22.276818 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:12:22.276822 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:12:22.276855 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:12:22.279453 (+ 2598us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:12:22.280261 (+ 808us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:12:22.280268 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:12:22.280270 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:12:22.280536 (+ 266us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:12:22.280998 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:12:22.281003 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:12:22.281006 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:12:22.281378 (+ 372us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:12:22.281596 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0923 20:12:22.281854 (+ 258us) server_negotiation.cc:299] Negotiation successful
0923 20:12:22.282121 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260923 20:12:22.286567 15637 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55492:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 20:12:22.293505 15637 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 20:12:22.332739 16108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:12:22.308168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.129:38865 (local address 127.0.0.1:44064)
0923 20:12:22.318926 (+ 10758us) negotiation.cc:107] Waiting for socket to connect
0923 20:12:22.319096 (+ 170us) client_negotiation.cc:174] Beginning negotiation
0923 20:12:22.319408 (+ 312us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:12:22.323074 (+ 3666us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:12:22.323081 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:12:22.323123 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:12:22.323626 (+ 503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:12:22.323641 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:12:22.326731 (+ 3090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:12:22.326737 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:12:22.329333 (+ 2596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:12:22.329355 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:12:22.331661 (+ 2306us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:12:22.331670 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:12:22.331917 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:12:22.331997 (+ 80us) client_negotiation.cc:769] Sending connection context
0923 20:12:22.332149 (+ 152us) client_negotiation.cc:240] Negotiation successful
0923 20:12:22.332359 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10314,"thread_start_us":13581,"threads_started":1}
I20260923 20:12:22.333940 16106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:12:22.308233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44064 (local address 127.1.170.129:38865)
0923 20:12:22.322482 (+ 14249us) server_negotiation.cc:206] Beginning negotiation
0923 20:12:22.322491 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 20:12:22.322514 (+ 23us) server_negotiation.cc:407] Connection header received
0923 20:12:22.322596 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:12:22.322602 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:12:22.322820 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:12:22.323834 (+ 1014us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:12:22.324082 (+ 248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:12:22.326558 (+ 2476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:12:22.329655 (+ 3097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:12:22.331424 (+ 1769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:12:22.331655 (+ 231us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:12:22.331933 (+ 278us) server_negotiation.cc:1035] Waiting for connection context
0923 20:12:22.333322 (+ 1389us) server_negotiation.cc:299] Negotiation successful
0923 20:12:22.333552 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14074,"thread_start_us":13859,"threads_started":1}
I20260923 20:12:22.335464 16109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:12:22.308749 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58324 (local address 127.1.170.130:37359)
0923 20:12:22.321605 (+ 12856us) server_negotiation.cc:206] Beginning negotiation
0923 20:12:22.321612 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:12:22.322185 (+ 573us) server_negotiation.cc:407] Connection header received
0923 20:12:22.322523 (+ 338us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:12:22.322534 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:12:22.322821 (+ 287us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:12:22.323001 (+ 180us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:12:22.325684 (+ 2683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:12:22.328918 (+ 3234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:12:22.332629 (+ 3711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:12:22.334235 (+ 1606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:12:22.334340 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:12:22.334483 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0923 20:12:22.334968 (+ 485us) server_negotiation.cc:299] Negotiation successful
0923 20:12:22.335150 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12694,"thread_start_us":12540,"threads_started":1}
I20260923 20:12:22.335464 16107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:12:22.308219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.130:37359 (local address 127.0.0.1:58324)
0923 20:12:22.322059 (+ 13840us) negotiation.cc:107] Waiting for socket to connect
0923 20:12:22.322084 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0923 20:12:22.322320 (+ 236us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:12:22.324675 (+ 2355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:12:22.324680 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:12:22.324705 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:12:22.325146 (+ 441us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:12:22.325292 (+ 146us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:12:22.329197 (+ 3905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:12:22.329203 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:12:22.332372 (+ 3169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:12:22.332384 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:12:22.334659 (+ 2275us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:12:22.334664 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:12:22.334794 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:12:22.334816 (+ 22us) client_negotiation.cc:769] Sending connection context
0923 20:12:22.334921 (+ 105us) client_negotiation.cc:240] Negotiation successful
0923 20:12:22.335070 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13637,"thread_start_us":11513,"threads_started":1}
I20260923 20:12:22.338501 16110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:12:22.322585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.131:40407 (local address 127.0.0.1:59164)
0923 20:12:22.329388 (+ 6803us) negotiation.cc:107] Waiting for socket to connect
0923 20:12:22.329415 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0923 20:12:22.329589 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:12:22.331030 (+ 1441us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:12:22.331035 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:12:22.331063 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:12:22.331479 (+ 416us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:12:22.331492 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:12:22.333467 (+ 1975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:12:22.333473 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:12:22.335601 (+ 2128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:12:22.335616 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:12:22.337700 (+ 2084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:12:22.337706 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:12:22.337896 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:12:22.337931 (+ 35us) client_negotiation.cc:769] Sending connection context
0923 20:12:22.338031 (+ 100us) client_negotiation.cc:240] Negotiation successful
0923 20:12:22.338170 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6583,"thread_start_us":7116,"threads_started":1}
I20260923 20:12:22.338186 15761 tablet_service.cc:1467] Processing CreateTablet for tablet a1d2d190cf8d43b3827b1b613a0dd401 (DEFAULT_TABLE table=test-table [id=a34063a94586449f92ae039737cca83a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 20:12:22.338940 15895 tablet_service.cc:1467] Processing CreateTablet for tablet a1d2d190cf8d43b3827b1b613a0dd401 (DEFAULT_TABLE table=test-table [id=a34063a94586449f92ae039737cca83a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 20:12:22.339386 16111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:12:22.322722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59164 (local address 127.1.170.131:40407)
0923 20:12:22.330488 (+ 7766us) server_negotiation.cc:206] Beginning negotiation
0923 20:12:22.330494 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:12:22.330524 (+ 30us) server_negotiation.cc:407] Connection header received
0923 20:12:22.330605 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:12:22.330611 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:12:22.330818 (+ 207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:12:22.331112 (+ 294us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:12:22.331720 (+ 608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:12:22.333285 (+ 1565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:12:22.335802 (+ 2517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:12:22.337203 (+ 1401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:12:22.337328 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:12:22.337554 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0923 20:12:22.338860 (+ 1306us) server_negotiation.cc:299] Negotiation successful
0923 20:12:22.339041 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7591,"thread_start_us":7437,"threads_started":1}
I20260923 20:12:22.339730 15761 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a1d2d190cf8d43b3827b1b613a0dd401. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:12:22.340088 15895 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a1d2d190cf8d43b3827b1b613a0dd401. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:12:22.343118 16028 tablet_service.cc:1467] Processing CreateTablet for tablet a1d2d190cf8d43b3827b1b613a0dd401 (DEFAULT_TABLE table=test-table [id=a34063a94586449f92ae039737cca83a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 20:12:22.344561 16028 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a1d2d190cf8d43b3827b1b613a0dd401. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:12:22.352890 16112 tablet_bootstrap.cc:492] T a1d2d190cf8d43b3827b1b613a0dd401 P 53297953fdab4397adf94a117c453bc4: Bootstrap starting.
I20260923 20:12:22.356412 16112 tablet_bootstrap.cc:654] T a1d2d190cf8d43b3827b1b613a0dd401 P 53297953fdab4397adf94a117c453bc4: Neither blocks nor log segments found. Creating new log.
I20260923 20:12:22.357467 16112 log.cc:824] T a1d2d190cf8d43b3827b1b613a0dd401 P 53297953fdab4397adf94a117c453bc4: Log is configured to *not* fsync() on all Append() calls
I20260923 20:12:22.358492 16113 tablet_bootstrap.cc:492] T a1d2d190cf8d43b3827b1b613a0dd401 P 7f5b750bad6042cda73694b87b237180: Bootstrap starting.
I20260923 20:12:22.361744 16115 tablet_bootstrap.cc:492] T a1d2d190cf8d43b3827b1b613a0dd401 P d06de688bc57487a9a34608cd7c70b6a: Bootstrap starting.
I20260923 20:12:22.362334 16112 tablet_bootstrap.cc:492] T a1d2d190cf8d43b3827b1b613a0dd401 P 53297953fdab4397adf94a117c453bc4: No bootstrap required, opened a new log
I20260923 20:12:22.362497 16113 tablet_bootstrap.cc:654] T a1d2d190cf8d43b3827b1b613a0dd401 P 7f5b750bad6042cda73694b87b237180: Neither blocks nor log segments found. Creating new log.
I20260923 20:12:22.362663 16112 ts_tablet_manager.cc:1397] T a1d2d190cf8d43b3827b1b613a0dd401 P 53297953fdab4397adf94a117c453bc4: Time spent bootstrapping tablet: real 0.010s user 0.009s sys 0.000s
I20260923 20:12:22.363765 16113 log.cc:824] T a1d2d190cf8d43b3827b1b613a0dd401 P 7f5b750bad6042cda73694b87b237180: Log is configured to *not* fsync() on all Append() calls
I20260923 20:12:22.365621 16115 tablet_bootstrap.cc:654] T a1d2d190cf8d43b3827b1b613a0dd401 P d06de688bc57487a9a34608cd7c70b6a: Neither blocks nor log segments found. Creating new log.
I20260923 20:12:22.367018 16115 log.cc:824] T a1d2d190cf8d43b3827b1b613a0dd401 P d06de688bc57487a9a34608cd7c70b6a: Log is configured to *not* fsync() on all Append() calls
I20260923 20:12:22.368645 16113 tablet_bootstrap.cc:492] T a1d2d190cf8d43b3827b1b613a0dd401 P 7f5b750bad6042cda73694b87b237180: No bootstrap required, opened a new log
I20260923 20:12:22.368965 16113 ts_tablet_manager.cc:1397] T a1d2d190cf8d43b3827b1b613a0dd401 P 7f5b750bad6042cda73694b87b237180: Time spent bootstrapping tablet: real 0.011s user 0.006s sys 0.003s
I20260923 20:12:22.372090 16112 raft_consensus.cc:348] T a1d2d190cf8d43b3827b1b613a0dd401 P 53297953fdab4397adf94a117c453bc4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f5b750bad6042cda73694b87b237180" member_type: VOTER last_known_addr { host: "127.1.170.129" port: 38865 } } peers { permanent_uuid: "53297953fdab4397adf94a117c453bc4" member_type: VOTER last_known_addr { host: "127.1.170.130" port: 37359 } } peers { permanent_uuid: "d06de688bc57487a9a34608cd7c70b6a" member_type: VOTER last_known_addr { host: "127.1.170.131" port: 40407 } }
I20260923 20:12:22.372566 16112 raft_consensus.cc:374] T a1d2d190cf8d43b3827b1b613a0dd401 P 53297953fdab4397adf94a117c453bc4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:12:22.372706 16112 raft_consensus.cc:729] T a1d2d190cf8d43b3827b1b613a0dd401 P 53297953fdab4397adf94a117c453bc4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53297953fdab4397adf94a117c453bc4, State: Initialized, Role: FOLLOWER
I20260923 20:12:22.372816 16115 tablet_bootstrap.cc:492] T a1d2d190cf8d43b3827b1b613a0dd401 P d06de688bc57487a9a34608cd7c70b6a: No bootstrap required, opened a new log
I20260923 20:12:22.373456 16115 ts_tablet_manager.cc:1397] T a1d2d190cf8d43b3827b1b613a0dd401 P d06de688bc57487a9a34608cd7c70b6a: Time spent bootstrapping tablet: real 0.012s user 0.010s sys 0.000s
I20260923 20:12:22.373457 16112 consensus_queue.cc:260] T a1d2d190cf8d43b3827b1b613a0dd401 P 53297953fdab4397adf94a117c453bc4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f5b750bad6042cda73694b87b237180" member_type: VOTER last_known_addr { host: "127.1.170.129" port: 38865 } } peers { permanent_uuid: "53297953fdab4397adf94a117c453bc4" member_type: VOTER last_known_addr { host: "127.1.170.130" port: 37359 } } peers { permanent_uuid: "d06de688bc57487a9a34608cd7c70b6a" member_type: VOTER last_known_addr { host: "127.1.170.131" port: 40407 } }
I20260923 20:12:22.374755 15960 heartbeater.cc:503] Master 127.1.170.190:43769 was elected leader, sending a full tablet report...
I20260923 20:12:22.375425 16112 ts_tablet_manager.cc:1428] T a1d2d190cf8d43b3827b1b613a0dd401 P 53297953fdab4397adf94a117c453bc4: Time spent starting tablet: real 0.013s user 0.008s sys 0.003s
I20260923 20:12:22.378907 16113 raft_consensus.cc:348] T a1d2d190cf8d43b3827b1b613a0dd401 P 7f5b750bad6042cda73694b87b237180 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f5b750bad6042cda73694b87b237180" member_type: VOTER last_known_addr { host: "127.1.170.129" port: 38865 } } peers { permanent_uuid: "53297953fdab4397adf94a117c453bc4" member_type: VOTER last_known_addr { host: "127.1.170.130" port: 37359 } } peers { permanent_uuid: "d06de688bc57487a9a34608cd7c70b6a" member_type: VOTER last_known_addr { host: "127.1.170.131" port: 40407 } }
I20260923 20:12:22.379330 16113 raft_consensus.cc:374] T a1d2d190cf8d43b3827b1b613a0dd401 P 7f5b750bad6042cda73694b87b237180 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:12:22.379451 16113 raft_consensus.cc:729] T a1d2d190cf8d43b3827b1b613a0dd401 P 7f5b750bad6042cda73694b87b237180 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f5b750bad6042cda73694b87b237180, State: Initialized, Role: FOLLOWER
I20260923 20:12:22.380129 16113 consensus_queue.cc:260] T a1d2d190cf8d43b3827b1b613a0dd401 P 7f5b750bad6042cda73694b87b237180 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f5b750bad6042cda73694b87b237180" member_type: VOTER last_known_addr { host: "127.1.170.129" port: 38865 } } peers { permanent_uuid: "53297953fdab4397adf94a117c453bc4" member_type: VOTER last_known_addr { host: "127.1.170.130" port: 37359 } } peers { permanent_uuid: "d06de688bc57487a9a34608cd7c70b6a" member_type: VOTER last_known_addr { host: "127.1.170.131" port: 40407 } }
I20260923 20:12:22.382130 16113 ts_tablet_manager.cc:1428] T a1d2d190cf8d43b3827b1b613a0dd401 P 7f5b750bad6042cda73694b87b237180: Time spent starting tablet: real 0.013s user 0.008s sys 0.005s
I20260923 20:12:22.385090 16115 raft_consensus.cc:348] T a1d2d190cf8d43b3827b1b613a0dd401 P d06de688bc57487a9a34608cd7c70b6a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f5b750bad6042cda73694b87b237180" member_type: VOTER last_known_addr { host: "127.1.170.129" port: 38865 } } peers { permanent_uuid: "53297953fdab4397adf94a117c453bc4" member_type: VOTER last_known_addr { host: "127.1.170.130" port: 37359 } } peers { permanent_uuid: "d06de688bc57487a9a34608cd7c70b6a" member_type: VOTER last_known_addr { host: "127.1.170.131" port: 40407 } }
I20260923 20:12:22.385495 16115 raft_consensus.cc:374] T a1d2d190cf8d43b3827b1b613a0dd401 P d06de688bc57487a9a34608cd7c70b6a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:12:22.385597 16115 raft_consensus.cc:729] T a1d2d190cf8d43b3827b1b613a0dd401 P d06de688bc57487a9a34608cd7c70b6a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d06de688bc57487a9a34608cd7c70b6a, State: Initialized, Role: FOLLOWER
I20260923 20:12:22.386294 16115 consensus_queue.cc:260] T a1d2d190cf8d43b3827b1b613a0dd401 P d06de688bc57487a9a34608cd7c70b6a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f5b750bad6042cda73694b87b237180" member_type: VOTER last_known_addr { host: "127.1.170.129" port: 38865 } } peers { permanent_uuid: "53297953fdab4397adf94a117c453bc4" member_type: VOTER last_known_addr { host: "127.1.170.130" port: 37359 } } peers { permanent_uuid: "d06de688bc57487a9a34608cd7c70b6a" member_type: VOTER last_known_addr { host: "127.1.170.131" port: 40407 } }
I20260923 20:12:22.387351 16093 heartbeater.cc:503] Master 127.1.170.190:43769 was elected leader, sending a full tablet report...
I20260923 20:12:22.387773 16115 ts_tablet_manager.cc:1428] T a1d2d190cf8d43b3827b1b613a0dd401 P d06de688bc57487a9a34608cd7c70b6a: Time spent starting tablet: real 0.014s user 0.012s sys 0.000s
W20260923 20:12:22.431489 15827 tablet.cc:2369] T a1d2d190cf8d43b3827b1b613a0dd401 P 7f5b750bad6042cda73694b87b237180: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260923 20:12:22.436296 15961 tablet.cc:2369] T a1d2d190cf8d43b3827b1b613a0dd401 P 53297953fdab4397adf94a117c453bc4: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260923 20:12:22.480019 16094 tablet.cc:2369] T a1d2d190cf8d43b3827b1b613a0dd401 P d06de688bc57487a9a34608cd7c70b6a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 20:12:22.482736 16119 raft_consensus.cc:482] T a1d2d190cf8d43b3827b1b613a0dd401 P 7f5b750bad6042cda73694b87b237180 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 20:12:22.482999 16119 raft_consensus.cc:504] T a1d2d190cf8d43b3827b1b613a0dd401 P 7f5b750bad6042cda73694b87b237180 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f5b750bad6042cda73694b87b237180" member_type: VOTER last_known_addr { host: "127.1.170.129" port: 38865 } } peers { permanent_uuid: "53297953fdab4397adf94a117c453bc4" member_type: VOTER last_known_addr { host: "127.1.170.130" port: 37359 } } peers { permanent_uuid: "d06de688bc57487a9a34608cd7c70b6a" member_type: VOTER last_known_addr { host: "127.1.170.131" port: 40407 } }
I20260923 20:12:22.484665 16119 leader_election.cc:288] T a1d2d190cf8d43b3827b1b613a0dd401 P 7f5b750bad6042cda73694b87b237180 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 53297953fdab4397adf94a117c453bc4 (127.1.170.130:37359), d06de688bc57487a9a34608cd7c70b6a (127.1.170.131:40407)
I20260923 20:12:22.493760 16111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:12:22.485079 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.170.129:50325 (local address 127.1.170.131:40407)
0923 20:12:22.485281 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0923 20:12:22.485288 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:12:22.486390 (+ 1102us) server_negotiation.cc:407] Connection header received
0923 20:12:22.486536 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:12:22.486539 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:12:22.486607 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:12:22.486674 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:12:22.487350 (+ 676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:12:22.488086 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:12:22.491134 (+ 3048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:12:22.492550 (+ 1416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:12:22.492737 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:12:22.492817 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0923 20:12:22.493327 (+ 510us) server_negotiation.cc:299] Negotiation successful
0923 20:12:22.493489 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260923 20:12:22.493760 16109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:12:22.485102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.170.129:40195 (local address 127.1.170.130:37359)
0923 20:12:22.485307 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0923 20:12:22.485313 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:12:22.486152 (+ 839us) server_negotiation.cc:407] Connection header received
0923 20:12:22.486325 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:12:22.486333 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:12:22.486419 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:12:22.486509 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:12:22.487433 (+ 924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:12:22.488133 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:12:22.491142 (+ 3009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:12:22.492570 (+ 1428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:12:22.492734 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:12:22.492817 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0923 20:12:22.493316 (+ 499us) server_negotiation.cc:299] Negotiation successful
0923 20:12:22.493465 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260923 20:12:22.493979 16122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:12:22.485253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.131:40407 (local address 127.1.170.129:50325)
0923 20:12:22.486285 (+ 1032us) negotiation.cc:107] Waiting for socket to connect
0923 20:12:22.486301 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0923 20:12:22.486424 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:12:22.486716 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:12:22.486720 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:12:22.486745 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:12:22.487089 (+ 344us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:12:22.487101 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:12:22.488212 (+ 1111us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:12:22.488220 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:12:22.490939 (+ 2719us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:12:22.490955 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:12:22.492854 (+ 1899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:12:22.492861 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:12:22.493151 (+ 290us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:12:22.493177 (+ 26us) client_negotiation.cc:769] Sending connection context
0923 20:12:22.493254 (+ 77us) client_negotiation.cc:240] Negotiation successful
0923 20:12:22.493455 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":904,"mutex_wait_us":220,"thread_start_us":527,"threads_started":1}
I20260923 20:12:22.494004 16121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:12:22.485073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.130:37359 (local address 127.1.170.129:40195)
0923 20:12:22.486034 (+ 961us) negotiation.cc:107] Waiting for socket to connect
0923 20:12:22.486054 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0923 20:12:22.486191 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:12:22.486802 (+ 611us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:12:22.486805 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:12:22.486843 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:12:22.487299 (+ 456us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:12:22.487307 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:12:22.488286 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:12:22.488290 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:12:22.490939 (+ 2649us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:12:22.490955 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:12:22.492866 (+ 1911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:12:22.492871 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:12:22.493091 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:12:22.493152 (+ 61us) client_negotiation.cc:769] Sending connection context
0923 20:12:22.493254 (+ 102us) client_negotiation.cc:240] Negotiation successful
0923 20:12:22.493455 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":722,"mutex_wait_us":223,"thread_start_us":488,"threads_started":1}
I20260923 20:12:22.495265 16048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1d2d190cf8d43b3827b1b613a0dd401" candidate_uuid: "7f5b750bad6042cda73694b87b237180" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d06de688bc57487a9a34608cd7c70b6a" is_pre_election: true
I20260923 20:12:22.495409 15915 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1d2d190cf8d43b3827b1b613a0dd401" candidate_uuid: "7f5b750bad6042cda73694b87b237180" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "53297953fdab4397adf94a117c453bc4" is_pre_election: true
I20260923 20:12:22.495816 16048 raft_consensus.cc:2454] T a1d2d190cf8d43b3827b1b613a0dd401 P d06de688bc57487a9a34608cd7c70b6a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7f5b750bad6042cda73694b87b237180 in term 0.
I20260923 20:12:22.495919 15915 raft_consensus.cc:2454] T a1d2d190cf8d43b3827b1b613a0dd401 P 53297953fdab4397adf94a117c453bc4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7f5b750bad6042cda73694b87b237180 in term 0.
I20260923 20:12:22.496595 15716 leader_election.cc:302] T a1d2d190cf8d43b3827b1b613a0dd401 P 7f5b750bad6042cda73694b87b237180 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 53297953fdab4397adf94a117c453bc4, 7f5b750bad6042cda73694b87b237180, d06de688bc57487a9a34608cd7c70b6a; no voters:
I20260923 20:12:22.497036 16119 raft_consensus.cc:2781] T a1d2d190cf8d43b3827b1b613a0dd401 P 7f5b750bad6042cda73694b87b237180 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 20:12:22.497239 16119 raft_consensus.cc:482] T a1d2d190cf8d43b3827b1b613a0dd401 P 7f5b750bad6042cda73694b87b237180 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 20:12:22.497364 16119 raft_consensus.cc:3037] T a1d2d190cf8d43b3827b1b613a0dd401 P 7f5b750bad6042cda73694b87b237180 [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:12:22.500102 16119 raft_consensus.cc:504] T a1d2d190cf8d43b3827b1b613a0dd401 P 7f5b750bad6042cda73694b87b237180 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f5b750bad6042cda73694b87b237180" member_type: VOTER last_known_addr { host: "127.1.170.129" port: 38865 } } peers { permanent_uuid: "53297953fdab4397adf94a117c453bc4" member_type: VOTER last_known_addr { host: "127.1.170.130" port: 37359 } } peers { permanent_uuid: "d06de688bc57487a9a34608cd7c70b6a" member_type: VOTER last_known_addr { host: "127.1.170.131" port: 40407 } }
I20260923 20:12:22.500761 16119 leader_election.cc:288] T a1d2d190cf8d43b3827b1b613a0dd401 P 7f5b750bad6042cda73694b87b237180 [CANDIDATE]: Term 1 election: Requested vote from peers 53297953fdab4397adf94a117c453bc4 (127.1.170.130:37359), d06de688bc57487a9a34608cd7c70b6a (127.1.170.131:40407)
I20260923 20:12:22.501257 16048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1d2d190cf8d43b3827b1b613a0dd401" candidate_uuid: "7f5b750bad6042cda73694b87b237180" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d06de688bc57487a9a34608cd7c70b6a"
I20260923 20:12:22.501266 15915 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1d2d190cf8d43b3827b1b613a0dd401" candidate_uuid: "7f5b750bad6042cda73694b87b237180" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "53297953fdab4397adf94a117c453bc4"
I20260923 20:12:22.501467 16048 raft_consensus.cc:3037] T a1d2d190cf8d43b3827b1b613a0dd401 P d06de688bc57487a9a34608cd7c70b6a [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:12:22.501478 15915 raft_consensus.cc:3037] T a1d2d190cf8d43b3827b1b613a0dd401 P 53297953fdab4397adf94a117c453bc4 [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:12:22.503714 16048 raft_consensus.cc:2454] T a1d2d190cf8d43b3827b1b613a0dd401 P d06de688bc57487a9a34608cd7c70b6a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7f5b750bad6042cda73694b87b237180 in term 1.
I20260923 20:12:22.503921 15915 raft_consensus.cc:2454] T a1d2d190cf8d43b3827b1b613a0dd401 P 53297953fdab4397adf94a117c453bc4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7f5b750bad6042cda73694b87b237180 in term 1.
I20260923 20:12:22.504165 15716 leader_election.cc:302] T a1d2d190cf8d43b3827b1b613a0dd401 P 7f5b750bad6042cda73694b87b237180 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7f5b750bad6042cda73694b87b237180, d06de688bc57487a9a34608cd7c70b6a; no voters:
I20260923 20:12:22.504535 16119 raft_consensus.cc:2781] T a1d2d190cf8d43b3827b1b613a0dd401 P 7f5b750bad6042cda73694b87b237180 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 20:12:22.504997 16119 raft_consensus.cc:686] T a1d2d190cf8d43b3827b1b613a0dd401 P 7f5b750bad6042cda73694b87b237180 [term 1 LEADER]: Becoming Leader. State: Replica: 7f5b750bad6042cda73694b87b237180, State: Running, Role: LEADER
I20260923 20:12:22.505529 16119 consensus_queue.cc:237] T a1d2d190cf8d43b3827b1b613a0dd401 P 7f5b750bad6042cda73694b87b237180 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f5b750bad6042cda73694b87b237180" member_type: VOTER last_known_addr { host: "127.1.170.129" port: 38865 } } peers { permanent_uuid: "53297953fdab4397adf94a117c453bc4" member_type: VOTER last_known_addr { host: "127.1.170.130" port: 37359 } } peers { permanent_uuid: "d06de688bc57487a9a34608cd7c70b6a" member_type: VOTER last_known_addr { host: "127.1.170.131" port: 40407 } }
I20260923 20:12:22.510262 15637 catalog_manager.cc:5495] T a1d2d190cf8d43b3827b1b613a0dd401 P 7f5b750bad6042cda73694b87b237180 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7f5b750bad6042cda73694b87b237180 (127.1.170.129). New cstate: current_term: 1 leader_uuid: "7f5b750bad6042cda73694b87b237180" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f5b750bad6042cda73694b87b237180" member_type: VOTER last_known_addr { host: "127.1.170.129" port: 38865 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "53297953fdab4397adf94a117c453bc4" member_type: VOTER last_known_addr { host: "127.1.170.130" port: 37359 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d06de688bc57487a9a34608cd7c70b6a" member_type: VOTER last_known_addr { host: "127.1.170.131" port: 40407 } health_report { overall_health: UNKNOWN } } }
I20260923 20:12:22.574726 16106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:12:22.567837 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44078 (local address 127.1.170.129:38865)
0923 20:12:22.568052 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0923 20:12:22.568058 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:12:22.568080 (+ 22us) server_negotiation.cc:407] Connection header received
0923 20:12:22.568310 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:12:22.568315 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:12:22.568434 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:12:22.568592 (+ 158us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 20:12:22.568993 (+ 401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:12:22.570098 (+ 1105us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:12:22.571995 (+ 1897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:12:22.572649 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:12:22.572772 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:12:22.573335 (+ 563us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 20:12:22.574063 (+ 728us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 20:12:22.574206 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0923 20:12:22.574319 (+ 113us) server_negotiation.cc:299] Negotiation successful
0923 20:12:22.574406 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260923 20:12:22.583702 16048 raft_consensus.cc:1261] T a1d2d190cf8d43b3827b1b613a0dd401 P d06de688bc57487a9a34608cd7c70b6a [term 1 FOLLOWER]: Refusing update from remote peer 7f5b750bad6042cda73694b87b237180: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 20:12:22.583703 15915 raft_consensus.cc:1261] T a1d2d190cf8d43b3827b1b613a0dd401 P 53297953fdab4397adf94a117c453bc4 [term 1 FOLLOWER]: Refusing update from remote peer 7f5b750bad6042cda73694b87b237180: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 20:12:22.584717 16123 consensus_queue.cc:1035] T a1d2d190cf8d43b3827b1b613a0dd401 P 7f5b750bad6042cda73694b87b237180 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d06de688bc57487a9a34608cd7c70b6a" member_type: VOTER last_known_addr { host: "127.1.170.131" port: 40407 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 20:12:22.585150 16119 consensus_queue.cc:1035] T a1d2d190cf8d43b3827b1b613a0dd401 P 7f5b750bad6042cda73694b87b237180 [LEADER]: Connected to new peer: Peer: permanent_uuid: "53297953fdab4397adf94a117c453bc4" member_type: VOTER last_known_addr { host: "127.1.170.130" port: 37359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 20:12:22.596530 16127 mvcc.cc:205] Tried to move back new op lower bound from 7332636027212910592 to 7332636026906771456. Current Snapshot: MvccSnapshot[applied={T|T < 7332636027212910592}]
I20260923 20:12:22.612979 15636 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55492:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 20:12:22.613443 15636 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55492:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 20:12:22.618141 15636 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e910e26b0d9645f0bca3a3198101aa5a: Sending DeleteTablet for 3 replicas of tablet a1d2d190cf8d43b3827b1b613a0dd401
I20260923 20:12:22.619378 15761 tablet_service.cc:1514] Processing DeleteTablet for tablet a1d2d190cf8d43b3827b1b613a0dd401 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 20:12:22 UTC) from {username='kudu', principal='kudu/127.1.170.190@KRBTEST.COM'} at 127.0.0.1:44064
I20260923 20:12:22.619419 15895 tablet_service.cc:1514] Processing DeleteTablet for tablet a1d2d190cf8d43b3827b1b613a0dd401 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 20:12:22 UTC) from {username='kudu', principal='kudu/127.1.170.190@KRBTEST.COM'} at 127.0.0.1:58324
I20260923 20:12:22.619565 1706 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskqUuku6/build/asan/bin/kudu pid 15698
I20260923 20:12:22.620539 16028 tablet_service.cc:1514] Processing DeleteTablet for tablet a1d2d190cf8d43b3827b1b613a0dd401 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 20:12:22 UTC) from {username='kudu', principal='kudu/127.1.170.190@KRBTEST.COM'} at 127.0.0.1:59164
I20260923 20:12:22.639864 16137 tablet_replica.cc:330] T a1d2d190cf8d43b3827b1b613a0dd401 P 53297953fdab4397adf94a117c453bc4: stopping tablet replica
I20260923 20:12:22.640079 16139 tablet_replica.cc:330] T a1d2d190cf8d43b3827b1b613a0dd401 P d06de688bc57487a9a34608cd7c70b6a: stopping tablet replica
I20260923 20:12:22.640405 16137 raft_consensus.cc:2229] T a1d2d190cf8d43b3827b1b613a0dd401 P 53297953fdab4397adf94a117c453bc4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 20:12:22.640429 16139 raft_consensus.cc:2229] T a1d2d190cf8d43b3827b1b613a0dd401 P d06de688bc57487a9a34608cd7c70b6a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 20:12:22.640951 16139 raft_consensus.cc:2258] T a1d2d190cf8d43b3827b1b613a0dd401 P d06de688bc57487a9a34608cd7c70b6a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 20:12:22.640951 16137 raft_consensus.cc:2258] T a1d2d190cf8d43b3827b1b613a0dd401 P 53297953fdab4397adf94a117c453bc4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 20:12:22.641752 16136 tablet_replica.cc:330] T a1d2d190cf8d43b3827b1b613a0dd401 P 7f5b750bad6042cda73694b87b237180: stopping tablet replica
I20260923 20:12:22.642810 16136 raft_consensus.cc:2229] T a1d2d190cf8d43b3827b1b613a0dd401 P 7f5b750bad6042cda73694b87b237180 [term 1 LEADER]: Raft consensus shutting down.
I20260923 20:12:22.643533 16136 raft_consensus.cc:2258] T a1d2d190cf8d43b3827b1b613a0dd401 P 7f5b750bad6042cda73694b87b237180 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 20:12:22.645164 16139 ts_tablet_manager.cc:1905] T a1d2d190cf8d43b3827b1b613a0dd401 P d06de688bc57487a9a34608cd7c70b6a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 20:12:22.645692 16136 ts_tablet_manager.cc:1905] T a1d2d190cf8d43b3827b1b613a0dd401 P 7f5b750bad6042cda73694b87b237180: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 23 20:12:22 dist-test-slave-th4g krb5kdc[15592](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194340, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.170.129@KRBTEST.COM
I20260923 20:12:22.648478 16137 ts_tablet_manager.cc:1905] T a1d2d190cf8d43b3827b1b613a0dd401 P 53297953fdab4397adf94a117c453bc4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 20:12:22.653024 16139 ts_tablet_manager.cc:1918] T a1d2d190cf8d43b3827b1b613a0dd401 P d06de688bc57487a9a34608cd7c70b6a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 20:12:22.653311 16139 log.cc:1196] T a1d2d190cf8d43b3827b1b613a0dd401 P d06de688bc57487a9a34608cd7c70b6a: Deleting WAL directory at /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194121577027-1706-0/minicluster-data/ts-2/wal/wals/a1d2d190cf8d43b3827b1b613a0dd401
I20260923 20:12:22.653347 16136 ts_tablet_manager.cc:1918] T a1d2d190cf8d43b3827b1b613a0dd401 P 7f5b750bad6042cda73694b87b237180: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 20:12:22.653563 16136 log.cc:1196] T a1d2d190cf8d43b3827b1b613a0dd401 P 7f5b750bad6042cda73694b87b237180: Deleting WAL directory at /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194121577027-1706-0/minicluster-data/ts-0/wal/wals/a1d2d190cf8d43b3827b1b613a0dd401
I20260923 20:12:22.654098 16139 ts_tablet_manager.cc:1939] T a1d2d190cf8d43b3827b1b613a0dd401 P d06de688bc57487a9a34608cd7c70b6a: Deleting consensus metadata
I20260923 20:12:22.654474 16136 ts_tablet_manager.cc:1939] T a1d2d190cf8d43b3827b1b613a0dd401 P 7f5b750bad6042cda73694b87b237180: Deleting consensus metadata
I20260923 20:12:22.656420 15623 catalog_manager.cc:4841] TS 7f5b750bad6042cda73694b87b237180 (127.1.170.129:38865): tablet a1d2d190cf8d43b3827b1b613a0dd401 (table test-table [id=a34063a94586449f92ae039737cca83a]) successfully deleted
I20260923 20:12:22.656780 15623 catalog_manager.cc:4841] TS d06de688bc57487a9a34608cd7c70b6a (127.1.170.131:40407): tablet a1d2d190cf8d43b3827b1b613a0dd401 (table test-table [id=a34063a94586449f92ae039737cca83a]) successfully deleted
I20260923 20:12:22.659482 16106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:12:22.621067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44080 (local address 127.1.170.129:38865)
0923 20:12:22.621287 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0923 20:12:22.621295 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 20:12:22.631257 (+ 9962us) server_negotiation.cc:407] Connection header received
0923 20:12:22.631850 (+ 593us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:12:22.631860 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:12:22.631992 (+ 132us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:12:22.632103 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:12:22.634891 (+ 2788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:12:22.636511 (+ 1620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:12:22.641391 (+ 4880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:12:22.642165 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:12:22.642278 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:12:22.651168 (+ 8890us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:12:22.651184 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:12:22.651203 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:12:22.651273 (+ 70us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:12:22.656195 (+ 4922us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:12:22.657184 (+ 989us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:12:22.657196 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:12:22.657202 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:12:22.657390 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:12:22.657827 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:12:22.657832 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:12:22.657836 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:12:22.658250 (+ 414us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:12:22.658564 (+ 314us) server_negotiation.cc:1035] Waiting for connection context
0923 20:12:22.658862 (+ 298us) server_negotiation.cc:299] Negotiation successful
0923 20:12:22.659109 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260923 20:12:22.659791 16137 ts_tablet_manager.cc:1918] T a1d2d190cf8d43b3827b1b613a0dd401 P 53297953fdab4397adf94a117c453bc4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 20:12:22.659955 16137 log.cc:1196] T a1d2d190cf8d43b3827b1b613a0dd401 P 53297953fdab4397adf94a117c453bc4: Deleting WAL directory at /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194121577027-1706-0/minicluster-data/ts-1/wal/wals/a1d2d190cf8d43b3827b1b613a0dd401
I20260923 20:12:22.660212 15821 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:12:22.660517 16137 ts_tablet_manager.cc:1939] T a1d2d190cf8d43b3827b1b613a0dd401 P 53297953fdab4397adf94a117c453bc4: Deleting consensus metadata
I20260923 20:12:22.662398 15621 catalog_manager.cc:4841] TS 53297953fdab4397adf94a117c453bc4 (127.1.170.130:37359): tablet a1d2d190cf8d43b3827b1b613a0dd401 (table test-table [id=a34063a94586449f92ae039737cca83a]) successfully deleted
I20260923 20:12:22.775382 1706 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqUuku6/build/asan/bin/kudu with pid 15698
I20260923 20:12:22.794729 1706 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskqUuku6/build/asan/bin/kudu pid 15832
Sep 23 20:12:22 dist-test-slave-th4g krb5kdc[15592](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194340, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.170.130@KRBTEST.COM
I20260923 20:12:22.810848 16109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:12:22.795582 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58338 (local address 127.1.170.130:37359)
0923 20:12:22.795788 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0923 20:12:22.795796 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 20:12:22.795842 (+ 46us) server_negotiation.cc:407] Connection header received
0923 20:12:22.796062 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:12:22.796069 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:12:22.796150 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:12:22.796267 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:12:22.797585 (+ 1318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:12:22.798528 (+ 943us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:12:22.799793 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:12:22.800192 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:12:22.800331 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:12:22.804741 (+ 4410us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:12:22.804756 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:12:22.804773 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:12:22.804813 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:12:22.807999 (+ 3186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:12:22.808733 (+ 734us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:12:22.808740 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:12:22.808743 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:12:22.808874 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:12:22.809256 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:12:22.809261 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:12:22.809263 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:12:22.809652 (+ 389us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:12:22.809993 (+ 341us) server_negotiation.cc:1035] Waiting for connection context
0923 20:12:22.810269 (+ 276us) server_negotiation.cc:299] Negotiation successful
0923 20:12:22.810490 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260923 20:12:22.811632 15955 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:12:22.903285 1706 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqUuku6/build/asan/bin/kudu with pid 15832
I20260923 20:12:22.920823 1706 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskqUuku6/build/asan/bin/kudu pid 15965
Sep 23 20:12:22 dist-test-slave-th4g krb5kdc[15592](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194340, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.170.131@KRBTEST.COM
I20260923 20:12:22.936926 16111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:12:22.921591 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59178 (local address 127.1.170.131:40407)
0923 20:12:22.921774 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0923 20:12:22.921780 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:12:22.921800 (+ 20us) server_negotiation.cc:407] Connection header received
0923 20:12:22.921946 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:12:22.921950 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:12:22.922050 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:12:22.922173 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:12:22.923481 (+ 1308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:12:22.924200 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:12:22.925497 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:12:22.926045 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:12:22.926174 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:12:22.930786 (+ 4612us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:12:22.930803 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:12:22.930820 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:12:22.930860 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:12:22.934059 (+ 3199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:12:22.934839 (+ 780us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:12:22.934846 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:12:22.934849 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:12:22.934992 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:12:22.935359 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:12:22.935365 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:12:22.935368 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:12:22.935726 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:12:22.936013 (+ 287us) server_negotiation.cc:1035] Waiting for connection context
0923 20:12:22.936347 (+ 334us) server_negotiation.cc:299] Negotiation successful
0923 20:12:22.936569 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260923 20:12:22.937595 16088 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:12:23.030537 1706 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqUuku6/build/asan/bin/kudu with pid 15965
I20260923 20:12:23.048394 1706 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskqUuku6/build/asan/bin/kudu pid 15605
I20260923 20:12:23.049629 15667 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:12:23.160660 1706 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqUuku6/build/asan/bin/kudu with pid 15605
I20260923 20:12:23.182451 1706 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqUuku6/build/asan/bin/kudu
/tmp/dist-test-taskqUuku6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194121577027-1706-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194121577027-1706-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194121577027-1706-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194121577027-1706-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.170.190:43769
--webserver_interface=127.1.170.190
--webserver_port=39057
--keytab_file=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194121577027-1706-0/krb5kdc/kudu_127.1.170.190.keytab
--principal=kudu/127.1.170.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.170.148:37943
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.170.190:43769
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194121577027-1706-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194121577027-1706-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194121577027-1706-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:12:23.509995 16144 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:12:23.510316 16144 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:12:23.510473 16144 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:12:23.517577 16144 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 20:12:23.517719 16144 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:12:23.517822 16144 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:12:23.517944 16144 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 20:12:23.518028 16144 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 20:12:23.527653 16144 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.170.148:37943
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194121577027-1706-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194121577027-1706-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.1.170.190:43769
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194121577027-1706-0/krb5kdc/kudu_127.1.170.190.keytab
--principal=kudu/127.1.170.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.170.190:43769
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194121577027-1706-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.170.190
--webserver_port=39057
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194121577027-1706-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:12:23.528970 16144 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:12:23.530257 16144 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:12:23.553005 16150 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:12:23.552974 16151 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:12:23.553027 16153 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:12:23.559106 16144 server_base.cc:1034] running on GCE node
I20260923 20:12:23.559973 16144 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:12:23.561528 16144 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:12:23.562776 16144 hybrid_clock.cc:648] HybridClock initialized: now 1790194343562715 us; error 60 us; skew 500 ppm
I20260923 20:12:23.563215 16144 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:12:23 dist-test-slave-th4g krb5kdc[15592](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194343, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.170.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:12:23.566733 16144 init.cc:377] Logged in from keytab as kudu/127.1.170.190@KRBTEST.COM (short username kudu)
I20260923 20:12:23.568410 16144 webserver.cc:460] Webserver started at http://127.1.170.190:39057/ using document root <none> and password file <none>
I20260923 20:12:23.569170 16144 fs_manager.cc:359] Metadata directory not provided
I20260923 20:12:23.569320 16144 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:12:23.575337 16144 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.000s sys 0.004s
I20260923 20:12:23.578263 16160 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:12:23.579654 16144 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260923 20:12:23.579831 16144 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194121577027-1706-0/minicluster-data/master-0/data,/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194121577027-1706-0/minicluster-data/master-0/wal
uuid: "e910e26b0d9645f0bca3a3198101aa5a"
format_stamp: "Formatted at 2026-09-23 20:12:20 on dist-test-slave-th4g"
server_key: "0322a97aa01b78f36a69b9b99ac35141"
server_key_iv: "469c0c9a2ee402c69b62f6ea74f31bac"
server_key_version: "encryptionkey@0"
I20260923 20:12:23.580881 16144 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194121577027-1706-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194121577027-1706-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194121577027-1706-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:12:23.600028 16144 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:12:23.604270 16144 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:12:23.604562 16144 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:12:23.624039 16144 rpc_server.cc:307] RPC server started. Bound to: 127.1.170.190:43769
I20260923 20:12:23.624061 16211 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.170.190:43769 every 8 connection(s)
I20260923 20:12:23.625715 16144 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194121577027-1706-0/minicluster-data/master-0/data/info.pb
I20260923 20:12:23.632557 16212 sys_catalog.cc:263] Verifying existing consensus state
I20260923 20:12:23.635380 1706 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqUuku6/build/asan/bin/kudu as pid 16144
I20260923 20:12:23.635865 16212 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e910e26b0d9645f0bca3a3198101aa5a: Bootstrap starting.
I20260923 20:12:23.636240 1706 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqUuku6/build/asan/bin/kudu
/tmp/dist-test-taskqUuku6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194121577027-1706-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194121577027-1706-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194121577027-1706-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194121577027-1706-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.170.129:38865
--local_ip_for_outbound_sockets=127.1.170.129
--tserver_master_addrs=127.1.170.190:43769
--webserver_port=42049
--webserver_interface=127.1.170.129
--keytab_file=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194121577027-1706-0/krb5kdc/kudu_127.1.170.129.keytab
--principal=kudu/127.1.170.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.170.148:37943
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194121577027-1706-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194121577027-1706-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194121577027-1706-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260923 20:12:23.661094 16212 log.cc:824] T 00000000000000000000000000000000 P e910e26b0d9645f0bca3a3198101aa5a: Log is configured to *not* fsync() on all Append() calls
I20260923 20:12:23.676276 16212 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e910e26b0d9645f0bca3a3198101aa5a: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260923 20:12:23.676878 16212 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e910e26b0d9645f0bca3a3198101aa5a: Bootstrap complete.
I20260923 20:12:23.690356 16212 raft_consensus.cc:348] T 00000000000000000000000000000000 P e910e26b0d9645f0bca3a3198101aa5a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e910e26b0d9645f0bca3a3198101aa5a" member_type: VOTER last_known_addr { host: "127.1.170.190" port: 43769 } }
I20260923 20:12:23.691603 16212 raft_consensus.cc:729] T 00000000000000000000000000000000 P e910e26b0d9645f0bca3a3198101aa5a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e910e26b0d9645f0bca3a3198101aa5a, State: Initialized, Role: FOLLOWER
I20260923 20:12:23.692306 16212 consensus_queue.cc:260] T 00000000000000000000000000000000 P e910e26b0d9645f0bca3a3198101aa5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e910e26b0d9645f0bca3a3198101aa5a" member_type: VOTER last_known_addr { host: "127.1.170.190" port: 43769 } }
I20260923 20:12:23.692591 16212 raft_consensus.cc:388] T 00000000000000000000000000000000 P e910e26b0d9645f0bca3a3198101aa5a [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 20:12:23.692716 16212 raft_consensus.cc:482] T 00000000000000000000000000000000 P e910e26b0d9645f0bca3a3198101aa5a [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 20:12:23.692904 16212 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e910e26b0d9645f0bca3a3198101aa5a [term 1 FOLLOWER]: Advancing to term 2
I20260923 20:12:23.698733 16212 raft_consensus.cc:504] T 00000000000000000000000000000000 P e910e26b0d9645f0bca3a3198101aa5a [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e910e26b0d9645f0bca3a3198101aa5a" member_type: VOTER last_known_addr { host: "127.1.170.190" port: 43769 } }
I20260923 20:12:23.699437 16212 leader_election.cc:302] T 00000000000000000000000000000000 P e910e26b0d9645f0bca3a3198101aa5a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e910e26b0d9645f0bca3a3198101aa5a; no voters:
I20260923 20:12:23.700160 16212 leader_election.cc:288] T 00000000000000000000000000000000 P e910e26b0d9645f0bca3a3198101aa5a [CANDIDATE]: Term 2 election: Requested vote from peers
I20260923 20:12:23.700328 16216 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e910e26b0d9645f0bca3a3198101aa5a [term 2 FOLLOWER]: Leader election won for term 2
I20260923 20:12:23.701808 16216 raft_consensus.cc:686] T 00000000000000000000000000000000 P e910e26b0d9645f0bca3a3198101aa5a [term 2 LEADER]: Becoming Leader. State: Replica: e910e26b0d9645f0bca3a3198101aa5a, State: Running, Role: LEADER
I20260923 20:12:23.702378 16216 consensus_queue.cc:237] T 00000000000000000000000000000000 P e910e26b0d9645f0bca3a3198101aa5a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e910e26b0d9645f0bca3a3198101aa5a" member_type: VOTER last_known_addr { host: "127.1.170.190" port: 43769 } }
I20260923 20:12:23.702721 16212 sys_catalog.cc:564] T 00000000000000000000000000000000 P e910e26b0d9645f0bca3a3198101aa5a [sys.catalog]: configured and running, proceeding with master startup.
I20260923 20:12:23.704658 16218 sys_catalog.cc:455] T 00000000000000000000000000000000 P e910e26b0d9645f0bca3a3198101aa5a [sys.catalog]: SysCatalogTable state changed. Reason: New leader e910e26b0d9645f0bca3a3198101aa5a. Latest consensus state: current_term: 2 leader_uuid: "e910e26b0d9645f0bca3a3198101aa5a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e910e26b0d9645f0bca3a3198101aa5a" member_type: VOTER last_known_addr { host: "127.1.170.190" port: 43769 } } }
I20260923 20:12:23.705025 16218 sys_catalog.cc:458] T 00000000000000000000000000000000 P e910e26b0d9645f0bca3a3198101aa5a [sys.catalog]: This master's current role is: LEADER
I20260923 20:12:23.705607 16217 sys_catalog.cc:455] T 00000000000000000000000000000000 P e910e26b0d9645f0bca3a3198101aa5a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "e910e26b0d9645f0bca3a3198101aa5a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e910e26b0d9645f0bca3a3198101aa5a" member_type: VOTER last_known_addr { host: "127.1.170.190" port: 43769 } } }
I20260923 20:12:23.705951 16217 sys_catalog.cc:458] T 00000000000000000000000000000000 P e910e26b0d9645f0bca3a3198101aa5a [sys.catalog]: This master's current role is: LEADER
I20260923 20:12:23.709069 16222 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 20:12:23.717474 16222 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 20:12:23.719027 16222 catalog_manager.cc:1259] Loaded cluster ID: 26a0561cd2ff469cb55904b448b78043
I20260923 20:12:23.719190 16222 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 20:12:23.722419 16222 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260923 20:12:23.741206 16222 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260923 20:12:23.753818 16222 catalog_manager.cc:1504] Loading token signing keys...
I20260923 20:12:23.755910 16222 catalog_manager.cc:5879] T 00000000000000000000000000000000 P e910e26b0d9645f0bca3a3198101aa5a: Loaded TSK: 0
I20260923 20:12:23.757484 16222 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 20:12:23.759707 16235 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
W20260923 20:12:24.011023 16214 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:12:24.011358 16214 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:12:24.011541 16214 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:12:24.018947 16214 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:12:24.019065 16214 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:12:24.019233 16214 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.170.129
I20260923 20:12:24.029072 16214 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.170.148:37943
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194121577027-1706-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194121577027-1706-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194121577027-1706-0/krb5kdc/kudu_127.1.170.129.keytab
--principal=kudu/127.1.170.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.170.129:38865
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194121577027-1706-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.170.129
--webserver_port=42049
--tserver_master_addrs=127.1.170.190:43769
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.170.129
--log_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194121577027-1706-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:12:24.030441 16214 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:12:24.031742 16214 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:12:24.046164 16241 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:12:24.046182 16240 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:12:24.046749 16243 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:12:24.047523 16214 server_base.cc:1034] running on GCE node
I20260923 20:12:24.048125 16214 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:12:24.049080 16214 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:12:24.050294 16214 hybrid_clock.cc:648] HybridClock initialized: now 1790194344050236 us; error 50 us; skew 500 ppm
I20260923 20:12:24.050616 16214 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:12:24 dist-test-slave-th4g krb5kdc[15592](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194344, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.170.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:12:24.058252 16214 init.cc:377] Logged in from keytab as kudu/127.1.170.129@KRBTEST.COM (short username kudu)
I20260923 20:12:24.059913 16214 webserver.cc:460] Webserver started at http://127.1.170.129:42049/ using document root <none> and password file <none>
I20260923 20:12:24.060592 16214 fs_manager.cc:359] Metadata directory not provided
I20260923 20:12:24.060729 16214 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:12:24.066171 16214 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.001s
I20260923 20:12:24.068941 16250 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:12:24.070367 16214 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 20:12:24.070544 16214 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194121577027-1706-0/minicluster-data/ts-0/data,/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194121577027-1706-0/minicluster-data/ts-0/wal
uuid: "7f5b750bad6042cda73694b87b237180"
format_stamp: "Formatted at 2026-09-23 20:12:21 on dist-test-slave-th4g"
server_key: "0ef88f7572d94c5b107314394af66393"
server_key_iv: "4fc80b00dd9f302d00610d10530c0ad8"
server_key_version: "encryptionkey@0"
I20260923 20:12:24.071507 16214 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194121577027-1706-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194121577027-1706-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194121577027-1706-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:12:24.087499 16214 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:12:24.091828 16214 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:12:24.092109 16214 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:12:24.093176 16214 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 20:12:24.094524 16214 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 20:12:24.094641 16214 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 20:12:24.094765 16214 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 20:12:24.094858 16214 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 20:12:24.128599 16214 rpc_server.cc:307] RPC server started. Bound to: 127.1.170.129:38865
I20260923 20:12:24.128684 16362 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.170.129:38865 every 8 connection(s)
I20260923 20:12:24.130044 16214 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194121577027-1706-0/minicluster-data/ts-0/data/info.pb
I20260923 20:12:24.133777 1706 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqUuku6/build/asan/bin/kudu as pid 16214
I20260923 20:12:24.134526 1706 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqUuku6/build/asan/bin/kudu
/tmp/dist-test-taskqUuku6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194121577027-1706-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194121577027-1706-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194121577027-1706-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194121577027-1706-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.170.130:37359
--local_ip_for_outbound_sockets=127.1.170.130
--tserver_master_addrs=127.1.170.190:43769
--webserver_port=41715
--webserver_interface=127.1.170.130
--keytab_file=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194121577027-1706-0/krb5kdc/kudu_127.1.170.130.keytab
--principal=kudu/127.1.170.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.170.148:37943
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194121577027-1706-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194121577027-1706-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194121577027-1706-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 23 20:12:24 dist-test-slave-th4g krb5kdc[15592](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194344, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.170.129@KRBTEST.COM for kudu/127.1.170.190@KRBTEST.COM
I20260923 20:12:24.154203 16365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:12:24.132880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.170.129:60367 (local address 127.1.170.190:43769)
0923 20:12:24.133587 (+ 707us) server_negotiation.cc:206] Beginning negotiation
0923 20:12:24.133606 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0923 20:12:24.134144 (+ 538us) server_negotiation.cc:407] Connection header received
0923 20:12:24.136250 (+ 2106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:12:24.136261 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:12:24.136639 (+ 378us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:12:24.136990 (+ 351us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:12:24.138501 (+ 1511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:12:24.139685 (+ 1184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:12:24.142153 (+ 2468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:12:24.142766 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:12:24.142914 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:12:24.146986 (+ 4072us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:12:24.146994 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:12:24.147007 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:12:24.147041 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:12:24.149959 (+ 2918us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:12:24.150658 (+ 699us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:12:24.150670 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:12:24.150675 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:12:24.150823 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:12:24.151243 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:12:24.151252 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:12:24.151254 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:12:24.151669 (+ 415us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:12:24.151871 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0923 20:12:24.152572 (+ 701us) server_negotiation.cc:299] Negotiation successful
0923 20:12:24.152942 (+ 370us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":480,"thread_start_us":295,"threads_started":1}
I20260923 20:12:24.154204 16366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:12:24.133089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.190:43769 (local address 127.1.170.129:60367)
0923 20:12:24.133984 (+ 895us) negotiation.cc:107] Waiting for socket to connect
0923 20:12:24.134021 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0923 20:12:24.135754 (+ 1733us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:12:24.137132 (+ 1378us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:12:24.137143 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:12:24.137721 (+ 578us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:12:24.138280 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:12:24.138296 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:12:24.139839 (+ 1543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:12:24.139845 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:12:24.142002 (+ 2157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:12:24.142020 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:12:24.143007 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:12:24.143017 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:12:24.143308 (+ 291us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:12:24.144044 (+ 736us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:12:24.144078 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:12:24.146721 (+ 2643us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:12:24.150091 (+ 3370us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:12:24.150101 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:12:24.150114 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:12:24.150516 (+ 402us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:12:24.150937 (+ 421us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:12:24.150942 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:12:24.150946 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:12:24.151122 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:12:24.151799 (+ 677us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:12:24.151806 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:12:24.152117 (+ 311us) client_negotiation.cc:769] Sending connection context
0923 20:12:24.152357 (+ 240us) client_negotiation.cc:240] Negotiation successful
0923 20:12:24.152888 (+ 531us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":601,"thread_start_us":364,"threads_started":1}
I20260923 20:12:24.157236 16363 heartbeater.cc:347] Connected to a master server at 127.1.170.190:43769
I20260923 20:12:24.157547 16363 heartbeater.cc:464] Registering TS with master...
I20260923 20:12:24.158362 16363 heartbeater.cc:511] Master 127.1.170.190:43769 requested a full tablet report, sending...
I20260923 20:12:24.160678 16177 ts_manager.cc:194] Registered new tserver with Master: 7f5b750bad6042cda73694b87b237180 (127.1.170.129:38865)
I20260923 20:12:24.162851 16177 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.170.129@KRBTEST.COM'} at 127.1.170.129:60367
W20260923 20:12:24.467327 16367 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:12:24.467646 16367 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:12:24.467761 16367 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:12:24.475417 16367 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:12:24.475546 16367 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:12:24.475714 16367 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.170.130
I20260923 20:12:24.486567 16367 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.170.148:37943
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194121577027-1706-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194121577027-1706-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194121577027-1706-0/krb5kdc/kudu_127.1.170.130.keytab
--principal=kudu/127.1.170.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.170.130:37359
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194121577027-1706-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.170.130
--webserver_port=41715
--tserver_master_addrs=127.1.170.190:43769
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.170.130
--log_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194121577027-1706-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:12:24.487983 16367 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:12:24.489295 16367 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:12:24.503360 16372 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:12:24.503269 16375 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:12:24.503316 16373 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:12:24.507527 16367 server_base.cc:1034] running on GCE node
I20260923 20:12:24.508265 16367 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:12:24.509362 16367 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:12:24.510726 16367 hybrid_clock.cc:648] HybridClock initialized: now 1790194344510647 us; error 67 us; skew 500 ppm
I20260923 20:12:24.511091 16367 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:12:24 dist-test-slave-th4g krb5kdc[15592](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194344, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.170.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:12:24.514780 16367 init.cc:377] Logged in from keytab as kudu/127.1.170.130@KRBTEST.COM (short username kudu)
I20260923 20:12:24.516460 16367 webserver.cc:460] Webserver started at http://127.1.170.130:41715/ using document root <none> and password file <none>
I20260923 20:12:24.517233 16367 fs_manager.cc:359] Metadata directory not provided
I20260923 20:12:24.517388 16367 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:12:24.522931 16367 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260923 20:12:24.525816 16382 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:12:24.527626 16367 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260923 20:12:24.527863 16367 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194121577027-1706-0/minicluster-data/ts-1/data,/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194121577027-1706-0/minicluster-data/ts-1/wal
uuid: "53297953fdab4397adf94a117c453bc4"
format_stamp: "Formatted at 2026-09-23 20:12:21 on dist-test-slave-th4g"
server_key: "3f95013455082ede0c61607aa4f14e3e"
server_key_iv: "546d26b1ce46b518a3cf9e42439abc2f"
server_key_version: "encryptionkey@0"
I20260923 20:12:24.528970 16367 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194121577027-1706-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194121577027-1706-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194121577027-1706-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:12:24.541678 16367 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:12:24.546394 16367 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:12:24.546679 16367 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:12:24.547726 16367 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 20:12:24.549237 16367 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 20:12:24.549356 16367 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 20:12:24.549481 16367 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 20:12:24.549572 16367 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 20:12:24.584255 16367 rpc_server.cc:307] RPC server started. Bound to: 127.1.170.130:37359
I20260923 20:12:24.584345 16494 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.170.130:37359 every 8 connection(s)
I20260923 20:12:24.585794 16367 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194121577027-1706-0/minicluster-data/ts-1/data/info.pb
I20260923 20:12:24.588577 1706 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqUuku6/build/asan/bin/kudu as pid 16367
I20260923 20:12:24.589289 1706 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqUuku6/build/asan/bin/kudu
/tmp/dist-test-taskqUuku6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194121577027-1706-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194121577027-1706-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194121577027-1706-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194121577027-1706-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.170.131:40407
--local_ip_for_outbound_sockets=127.1.170.131
--tserver_master_addrs=127.1.170.190:43769
--webserver_port=34681
--webserver_interface=127.1.170.131
--keytab_file=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194121577027-1706-0/krb5kdc/kudu_127.1.170.131.keytab
--principal=kudu/127.1.170.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.170.148:37943
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194121577027-1706-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194121577027-1706-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194121577027-1706-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 23 20:12:24 dist-test-slave-th4g krb5kdc[15592](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194344, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.170.130@KRBTEST.COM for kudu/127.1.170.190@KRBTEST.COM
I20260923 20:12:24.607640 16365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:12:24.588450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.170.130:44627 (local address 127.1.170.190:43769)
0923 20:12:24.588666 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0923 20:12:24.588673 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:12:24.589961 (+ 1288us) server_negotiation.cc:407] Connection header received
0923 20:12:24.591823 (+ 1862us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:12:24.591832 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:12:24.591938 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:12:24.592068 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:12:24.593618 (+ 1550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:12:24.594908 (+ 1290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:12:24.596545 (+ 1637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:12:24.596975 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:12:24.597117 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:12:24.601408 (+ 4291us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:12:24.601414 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:12:24.601418 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:12:24.601456 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:12:24.604335 (+ 2879us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:12:24.605133 (+ 798us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:12:24.605144 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:12:24.605148 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:12:24.605276 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:12:24.605732 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:12:24.605738 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:12:24.605742 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:12:24.606100 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:12:24.606264 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0923 20:12:24.607006 (+ 742us) server_negotiation.cc:299] Negotiation successful
0923 20:12:24.607263 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260923 20:12:24.608771 16497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:12:24.588969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.190:43769 (local address 127.1.170.130:44627)
0923 20:12:24.589750 (+ 781us) negotiation.cc:107] Waiting for socket to connect
0923 20:12:24.589810 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0923 20:12:24.591542 (+ 1732us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:12:24.592275 (+ 733us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:12:24.592283 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:12:24.592786 (+ 503us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:12:24.593416 (+ 630us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:12:24.593437 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:12:24.595110 (+ 1673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:12:24.595114 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:12:24.596369 (+ 1255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:12:24.596381 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:12:24.597211 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:12:24.597217 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:12:24.597536 (+ 319us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:12:24.598346 (+ 810us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:12:24.598380 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:12:24.601171 (+ 2791us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:12:24.604496 (+ 3325us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:12:24.604505 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:12:24.604524 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:12:24.604988 (+ 464us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:12:24.605398 (+ 410us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:12:24.605403 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:12:24.605408 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:12:24.605612 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:12:24.606243 (+ 631us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:12:24.606252 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:12:24.606624 (+ 372us) client_negotiation.cc:769] Sending connection context
0923 20:12:24.606910 (+ 286us) client_negotiation.cc:240] Negotiation successful
0923 20:12:24.607485 (+ 575us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":511,"thread_start_us":332,"threads_started":1}
I20260923 20:12:24.610335 16495 heartbeater.cc:347] Connected to a master server at 127.1.170.190:43769
I20260923 20:12:24.610680 16495 heartbeater.cc:464] Registering TS with master...
I20260923 20:12:24.611459 16495 heartbeater.cc:511] Master 127.1.170.190:43769 requested a full tablet report, sending...
I20260923 20:12:24.613024 16177 ts_manager.cc:194] Registered new tserver with Master: 53297953fdab4397adf94a117c453bc4 (127.1.170.130:37359)
I20260923 20:12:24.614221 16177 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.170.130@KRBTEST.COM'} at 127.1.170.130:44627
W20260923 20:12:24.911607 16498 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:12:24.911907 16498 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:12:24.912009 16498 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:12:24.918919 16498 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:12:24.919018 16498 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:12:24.919133 16498 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.170.131
I20260923 20:12:24.928244 16498 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.170.148:37943
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194121577027-1706-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194121577027-1706-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194121577027-1706-0/krb5kdc/kudu_127.1.170.131.keytab
--principal=kudu/127.1.170.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.170.131:40407
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194121577027-1706-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.170.131
--webserver_port=34681
--tserver_master_addrs=127.1.170.190:43769
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.170.131
--log_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194121577027-1706-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:12:24.929394 16498 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:12:24.930696 16498 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:12:24.943475 16504 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:12:24.943516 16506 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:12:24.943544 16503 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:12:24.944146 16498 server_base.cc:1034] running on GCE node
I20260923 20:12:24.944708 16498 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:12:24.945641 16498 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:12:24.946849 16498 hybrid_clock.cc:648] HybridClock initialized: now 1790194344946794 us; error 74 us; skew 500 ppm
I20260923 20:12:24.947162 16498 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:12:24 dist-test-slave-th4g krb5kdc[15592](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194344, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.170.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:12:24.950206 16498 init.cc:377] Logged in from keytab as kudu/127.1.170.131@KRBTEST.COM (short username kudu)
I20260923 20:12:24.951541 16498 webserver.cc:460] Webserver started at http://127.1.170.131:34681/ using document root <none> and password file <none>
I20260923 20:12:24.952198 16498 fs_manager.cc:359] Metadata directory not provided
I20260923 20:12:24.952340 16498 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:12:24.957297 16498 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.001s
I20260923 20:12:24.959914 16513 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:12:24.961140 16498 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260923 20:12:24.961309 16498 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194121577027-1706-0/minicluster-data/ts-2/data,/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194121577027-1706-0/minicluster-data/ts-2/wal
uuid: "d06de688bc57487a9a34608cd7c70b6a"
format_stamp: "Formatted at 2026-09-23 20:12:22 on dist-test-slave-th4g"
server_key: "6f84fe378bd8c0618a4fe5ebbe74fe2e"
server_key_iv: "344643454d475f00f5172ce964fc04f8"
server_key_version: "encryptionkey@0"
I20260923 20:12:24.962385 16498 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194121577027-1706-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194121577027-1706-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194121577027-1706-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:12:24.988590 16498 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:12:24.992722 16498 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:12:24.993026 16498 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:12:24.994060 16498 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 20:12:24.995362 16498 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 20:12:24.995479 16498 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 20:12:24.995609 16498 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 20:12:24.995695 16498 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 20:12:25.031993 16498 rpc_server.cc:307] RPC server started. Bound to: 127.1.170.131:40407
I20260923 20:12:25.032044 16625 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.170.131:40407 every 8 connection(s)
I20260923 20:12:25.033422 16498 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194121577027-1706-0/minicluster-data/ts-2/data/info.pb
I20260923 20:12:25.043309 1706 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqUuku6/build/asan/bin/kudu as pid 16498
Sep 23 20:12:25 dist-test-slave-th4g krb5kdc[15592](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194344, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.170.131@KRBTEST.COM for kudu/127.1.170.190@KRBTEST.COM
I20260923 20:12:25.054131 16365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:12:25.035895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.170.131:42193 (local address 127.1.170.190:43769)
0923 20:12:25.036073 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0923 20:12:25.036078 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 20:12:25.037137 (+ 1059us) server_negotiation.cc:407] Connection header received
0923 20:12:25.039009 (+ 1872us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:12:25.039015 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:12:25.039102 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:12:25.039209 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:12:25.040736 (+ 1527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:12:25.041523 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:12:25.043223 (+ 1700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:12:25.043707 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:12:25.043813 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:12:25.047667 (+ 3854us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:12:25.047672 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:12:25.047676 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:12:25.047706 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:12:25.050822 (+ 3116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:12:25.051715 (+ 893us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:12:25.051723 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:12:25.051728 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:12:25.051875 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:12:25.052308 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:12:25.052312 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:12:25.052316 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:12:25.052688 (+ 372us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:12:25.052848 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0923 20:12:25.053565 (+ 717us) server_negotiation.cc:299] Negotiation successful
0923 20:12:25.053782 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260923 20:12:25.055362 16628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:12:25.036262 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.190:43769 (local address 127.1.170.131:42193)
0923 20:12:25.036943 (+ 681us) negotiation.cc:107] Waiting for socket to connect
0923 20:12:25.037020 (+ 77us) client_negotiation.cc:174] Beginning negotiation
0923 20:12:25.038754 (+ 1734us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:12:25.039422 (+ 668us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:12:25.039429 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:12:25.039929 (+ 500us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:12:25.040540 (+ 611us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:12:25.040568 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:12:25.041667 (+ 1099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:12:25.041671 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:12:25.043062 (+ 1391us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:12:25.043080 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:12:25.043923 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:12:25.043928 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:12:25.044198 (+ 270us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:12:25.044894 (+ 696us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:12:25.044924 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:12:25.047467 (+ 2543us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:12:25.050962 (+ 3495us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:12:25.050969 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:12:25.050981 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:12:25.051584 (+ 603us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:12:25.051977 (+ 393us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:12:25.051980 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:12:25.051982 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:12:25.052188 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:12:25.052831 (+ 643us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:12:25.052837 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:12:25.053192 (+ 355us) client_negotiation.cc:769] Sending connection context
0923 20:12:25.053416 (+ 224us) client_negotiation.cc:240] Negotiation successful
0923 20:12:25.053946 (+ 530us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":446,"thread_start_us":257,"threads_started":1}
I20260923 20:12:25.056788 16626 heartbeater.cc:347] Connected to a master server at 127.1.170.190:43769
I20260923 20:12:25.057135 16626 heartbeater.cc:464] Registering TS with master...
I20260923 20:12:25.057924 16626 heartbeater.cc:511] Master 127.1.170.190:43769 requested a full tablet report, sending...
I20260923 20:12:25.059572 16177 ts_manager.cc:194] Registered new tserver with Master: d06de688bc57487a9a34608cd7c70b6a (127.1.170.131:40407)
I20260923 20:12:25.059758 16629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:12:25.044230 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41952 (local address 127.1.170.190:43769)
0923 20:12:25.045067 (+ 837us) server_negotiation.cc:206] Beginning negotiation
0923 20:12:25.045074 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:12:25.045378 (+ 304us) server_negotiation.cc:407] Connection header received
0923 20:12:25.045506 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:12:25.045511 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:12:25.045597 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:12:25.045707 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:12:25.047193 (+ 1486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:12:25.048087 (+ 894us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:12:25.050139 (+ 2052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:12:25.050586 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:12:25.050690 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:12:25.053733 (+ 3043us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:12:25.053741 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:12:25.053745 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:12:25.053781 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:12:25.057083 (+ 3302us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:12:25.057775 (+ 692us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:12:25.057784 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:12:25.057787 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:12:25.057964 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:12:25.058325 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:12:25.058331 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:12:25.058336 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:12:25.058671 (+ 335us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:12:25.058888 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0923 20:12:25.059236 (+ 348us) server_negotiation.cc:299] Negotiation successful
0923 20:12:25.059402 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":707,"thread_start_us":555,"threads_started":1}
I20260923 20:12:25.060588 16177 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.170.131@KRBTEST.COM'} at 127.1.170.131:42193
I20260923 20:12:25.062278 1706 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260923 20:12:25.068727 16629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:12:25.063181 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41956 (local address 127.1.170.190:43769)
0923 20:12:25.063344 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0923 20:12:25.063348 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 20:12:25.063783 (+ 435us) server_negotiation.cc:407] Connection header received
0923 20:12:25.063948 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:12:25.063951 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:12:25.064063 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:12:25.064177 (+ 114us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 20:12:25.064607 (+ 430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:12:25.065260 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:12:25.066920 (+ 1660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:12:25.067331 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:12:25.067418 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:12:25.067851 (+ 433us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 20:12:25.068280 (+ 429us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 20:12:25.068386 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0923 20:12:25.068467 (+ 81us) server_negotiation.cc:299] Negotiation successful
0923 20:12:25.068524 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260923 20:12:25.069591 16177 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:41956:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 20:12:25.077054 16177 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 20:12:25.100656 16638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:12:25.089479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34724 (local address 127.1.170.131:40407)
0923 20:12:25.091822 (+ 2343us) server_negotiation.cc:206] Beginning negotiation
0923 20:12:25.091835 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0923 20:12:25.091875 (+ 40us) server_negotiation.cc:407] Connection header received
0923 20:12:25.091975 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:12:25.091982 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:12:25.092249 (+ 267us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:12:25.093292 (+ 1043us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:12:25.093786 (+ 494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:12:25.095201 (+ 1415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:12:25.098235 (+ 3034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:12:25.099583 (+ 1348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:12:25.099714 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:12:25.099920 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0923 20:12:25.100164 (+ 244us) server_negotiation.cc:299] Negotiation successful
0923 20:12:25.100323 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2122,"thread_start_us":1926,"threads_started":1}
I20260923 20:12:25.100705 16636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:12:25.089050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.131:40407 (local address 127.0.0.1:34724)
0923 20:12:25.089980 (+ 930us) negotiation.cc:107] Waiting for socket to connect
0923 20:12:25.090020 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0923 20:12:25.090198 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:12:25.092516 (+ 2318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:12:25.092524 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:12:25.092557 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:12:25.093054 (+ 497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:12:25.093067 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:12:25.095325 (+ 2258us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:12:25.095333 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:12:25.098040 (+ 2707us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:12:25.098057 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:12:25.099780 (+ 1723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:12:25.099786 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:12:25.099979 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:12:25.100015 (+ 36us) client_negotiation.cc:769] Sending connection context
0923 20:12:25.100121 (+ 106us) client_negotiation.cc:240] Negotiation successful
0923 20:12:25.100248 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":613,"mutex_wait_us":156,"thread_start_us":612,"threads_started":1}
I20260923 20:12:25.100864 16641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:12:25.092191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41136 (local address 127.1.170.129:38865)
0923 20:12:25.093096 (+ 905us) server_negotiation.cc:206] Beginning negotiation
0923 20:12:25.093104 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 20:12:25.093128 (+ 24us) server_negotiation.cc:407] Connection header received
0923 20:12:25.093221 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:12:25.093227 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:12:25.093430 (+ 203us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:12:25.093623 (+ 193us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:12:25.094290 (+ 667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:12:25.095668 (+ 1378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:12:25.097750 (+ 2082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:12:25.099267 (+ 1517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:12:25.099404 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:12:25.099602 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0923 20:12:25.100353 (+ 751us) server_negotiation.cc:299] Negotiation successful
0923 20:12:25.100534 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":742,"thread_start_us":597,"threads_started":1}
I20260923 20:12:25.101279 16637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:12:25.089231 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.130:37359 (local address 127.0.0.1:49694)
0923 20:12:25.090923 (+ 1692us) negotiation.cc:107] Waiting for socket to connect
0923 20:12:25.090946 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0923 20:12:25.091715 (+ 769us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:12:25.092826 (+ 1111us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:12:25.092831 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:12:25.092854 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:12:25.093227 (+ 373us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:12:25.093235 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:12:25.095484 (+ 2249us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:12:25.095498 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:12:25.098468 (+ 2970us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:12:25.098484 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:12:25.100417 (+ 1933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:12:25.100424 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:12:25.100606 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:12:25.100646 (+ 40us) client_negotiation.cc:769] Sending connection context
0923 20:12:25.100741 (+ 95us) client_negotiation.cc:240] Negotiation successful
0923 20:12:25.100866 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1532,"mutex_wait_us":91,"thread_start_us":1353,"threads_started":1}
I20260923 20:12:25.101331 16640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:12:25.091722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.129:38865 (local address 127.0.0.1:41136)
0923 20:12:25.092461 (+ 739us) negotiation.cc:107] Waiting for socket to connect
0923 20:12:25.092478 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0923 20:12:25.092577 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:12:25.093622 (+ 1045us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:12:25.093626 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:12:25.093648 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:12:25.094080 (+ 432us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:12:25.094094 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:12:25.095793 (+ 1699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:12:25.095801 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:12:25.097599 (+ 1798us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:12:25.097611 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:12:25.099475 (+ 1864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:12:25.099482 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:12:25.099680 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:12:25.099733 (+ 53us) client_negotiation.cc:769] Sending connection context
0923 20:12:25.099866 (+ 133us) client_negotiation.cc:240] Negotiation successful
0923 20:12:25.100073 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":603,"mutex_wait_us":56,"spinlock_wait_cycles":4480,"thread_start_us":616,"threads_started":1}
I20260923 20:12:25.102094 16639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:12:25.089657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49694 (local address 127.1.170.130:37359)
0923 20:12:25.092189 (+ 2532us) server_negotiation.cc:206] Beginning negotiation
0923 20:12:25.092197 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 20:12:25.092227 (+ 30us) server_negotiation.cc:407] Connection header received
0923 20:12:25.092320 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:12:25.092330 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:12:25.092557 (+ 227us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:12:25.093420 (+ 863us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:12:25.093713 (+ 293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:12:25.095342 (+ 1629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:12:25.098766 (+ 3424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:12:25.099930 (+ 1164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:12:25.100057 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:12:25.100260 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0923 20:12:25.101495 (+ 1235us) server_negotiation.cc:299] Negotiation successful
0923 20:12:25.101683 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2370,"thread_start_us":2251,"threads_started":1}
I20260923 20:12:25.105022 16561 tablet_service.cc:1467] Processing CreateTablet for tablet 691e285e2e31466e992aae1e9af38189 (DEFAULT_TABLE table=test-table [id=06b14acadb5345d8a622475de12fcabe]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 20:12:25.105022 16298 tablet_service.cc:1467] Processing CreateTablet for tablet 691e285e2e31466e992aae1e9af38189 (DEFAULT_TABLE table=test-table [id=06b14acadb5345d8a622475de12fcabe]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 20:12:25.106438 16561 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 691e285e2e31466e992aae1e9af38189. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:12:25.106441 16298 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 691e285e2e31466e992aae1e9af38189. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:12:25.107797 16430 tablet_service.cc:1467] Processing CreateTablet for tablet 691e285e2e31466e992aae1e9af38189 (DEFAULT_TABLE table=test-table [id=06b14acadb5345d8a622475de12fcabe]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 20:12:25.108814 16430 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 691e285e2e31466e992aae1e9af38189. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:12:25.120579 16642 tablet_bootstrap.cc:492] T 691e285e2e31466e992aae1e9af38189 P 53297953fdab4397adf94a117c453bc4: Bootstrap starting.
I20260923 20:12:25.122825 16643 tablet_bootstrap.cc:492] T 691e285e2e31466e992aae1e9af38189 P 7f5b750bad6042cda73694b87b237180: Bootstrap starting.
I20260923 20:12:25.126085 16642 tablet_bootstrap.cc:654] T 691e285e2e31466e992aae1e9af38189 P 53297953fdab4397adf94a117c453bc4: Neither blocks nor log segments found. Creating new log.
I20260923 20:12:25.126210 16643 tablet_bootstrap.cc:654] T 691e285e2e31466e992aae1e9af38189 P 7f5b750bad6042cda73694b87b237180: Neither blocks nor log segments found. Creating new log.
I20260923 20:12:25.126670 16644 tablet_bootstrap.cc:492] T 691e285e2e31466e992aae1e9af38189 P d06de688bc57487a9a34608cd7c70b6a: Bootstrap starting.
I20260923 20:12:25.127519 16643 log.cc:824] T 691e285e2e31466e992aae1e9af38189 P 7f5b750bad6042cda73694b87b237180: Log is configured to *not* fsync() on all Append() calls
I20260923 20:12:25.127529 16642 log.cc:824] T 691e285e2e31466e992aae1e9af38189 P 53297953fdab4397adf94a117c453bc4: Log is configured to *not* fsync() on all Append() calls
I20260923 20:12:25.130226 16644 tablet_bootstrap.cc:654] T 691e285e2e31466e992aae1e9af38189 P d06de688bc57487a9a34608cd7c70b6a: Neither blocks nor log segments found. Creating new log.
I20260923 20:12:25.131321 16644 log.cc:824] T 691e285e2e31466e992aae1e9af38189 P d06de688bc57487a9a34608cd7c70b6a: Log is configured to *not* fsync() on all Append() calls
I20260923 20:12:25.137099 16642 tablet_bootstrap.cc:492] T 691e285e2e31466e992aae1e9af38189 P 53297953fdab4397adf94a117c453bc4: No bootstrap required, opened a new log
I20260923 20:12:25.137388 16642 ts_tablet_manager.cc:1397] T 691e285e2e31466e992aae1e9af38189 P 53297953fdab4397adf94a117c453bc4: Time spent bootstrapping tablet: real 0.017s user 0.006s sys 0.006s
I20260923 20:12:25.137933 16643 tablet_bootstrap.cc:492] T 691e285e2e31466e992aae1e9af38189 P 7f5b750bad6042cda73694b87b237180: No bootstrap required, opened a new log
I20260923 20:12:25.138197 16643 ts_tablet_manager.cc:1397] T 691e285e2e31466e992aae1e9af38189 P 7f5b750bad6042cda73694b87b237180: Time spent bootstrapping tablet: real 0.016s user 0.005s sys 0.008s
I20260923 20:12:25.141408 16644 tablet_bootstrap.cc:492] T 691e285e2e31466e992aae1e9af38189 P d06de688bc57487a9a34608cd7c70b6a: No bootstrap required, opened a new log
I20260923 20:12:25.141669 16644 ts_tablet_manager.cc:1397] T 691e285e2e31466e992aae1e9af38189 P d06de688bc57487a9a34608cd7c70b6a: Time spent bootstrapping tablet: real 0.015s user 0.007s sys 0.004s
I20260923 20:12:25.148164 16642 raft_consensus.cc:348] T 691e285e2e31466e992aae1e9af38189 P 53297953fdab4397adf94a117c453bc4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d06de688bc57487a9a34608cd7c70b6a" member_type: VOTER last_known_addr { host: "127.1.170.131" port: 40407 } } peers { permanent_uuid: "53297953fdab4397adf94a117c453bc4" member_type: VOTER last_known_addr { host: "127.1.170.130" port: 37359 } } peers { permanent_uuid: "7f5b750bad6042cda73694b87b237180" member_type: VOTER last_known_addr { host: "127.1.170.129" port: 38865 } }
I20260923 20:12:25.148622 16642 raft_consensus.cc:374] T 691e285e2e31466e992aae1e9af38189 P 53297953fdab4397adf94a117c453bc4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:12:25.148731 16642 raft_consensus.cc:729] T 691e285e2e31466e992aae1e9af38189 P 53297953fdab4397adf94a117c453bc4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53297953fdab4397adf94a117c453bc4, State: Initialized, Role: FOLLOWER
I20260923 20:12:25.148993 16643 raft_consensus.cc:348] T 691e285e2e31466e992aae1e9af38189 P 7f5b750bad6042cda73694b87b237180 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d06de688bc57487a9a34608cd7c70b6a" member_type: VOTER last_known_addr { host: "127.1.170.131" port: 40407 } } peers { permanent_uuid: "53297953fdab4397adf94a117c453bc4" member_type: VOTER last_known_addr { host: "127.1.170.130" port: 37359 } } peers { permanent_uuid: "7f5b750bad6042cda73694b87b237180" member_type: VOTER last_known_addr { host: "127.1.170.129" port: 38865 } }
I20260923 20:12:25.149400 16643 raft_consensus.cc:374] T 691e285e2e31466e992aae1e9af38189 P 7f5b750bad6042cda73694b87b237180 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:12:25.149516 16643 raft_consensus.cc:729] T 691e285e2e31466e992aae1e9af38189 P 7f5b750bad6042cda73694b87b237180 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f5b750bad6042cda73694b87b237180, State: Initialized, Role: FOLLOWER
I20260923 20:12:25.149451 16642 consensus_queue.cc:260] T 691e285e2e31466e992aae1e9af38189 P 53297953fdab4397adf94a117c453bc4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d06de688bc57487a9a34608cd7c70b6a" member_type: VOTER last_known_addr { host: "127.1.170.131" port: 40407 } } peers { permanent_uuid: "53297953fdab4397adf94a117c453bc4" member_type: VOTER last_known_addr { host: "127.1.170.130" port: 37359 } } peers { permanent_uuid: "7f5b750bad6042cda73694b87b237180" member_type: VOTER last_known_addr { host: "127.1.170.129" port: 38865 } }
I20260923 20:12:25.150307 16643 consensus_queue.cc:260] T 691e285e2e31466e992aae1e9af38189 P 7f5b750bad6042cda73694b87b237180 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d06de688bc57487a9a34608cd7c70b6a" member_type: VOTER last_known_addr { host: "127.1.170.131" port: 40407 } } peers { permanent_uuid: "53297953fdab4397adf94a117c453bc4" member_type: VOTER last_known_addr { host: "127.1.170.130" port: 37359 } } peers { permanent_uuid: "7f5b750bad6042cda73694b87b237180" member_type: VOTER last_known_addr { host: "127.1.170.129" port: 38865 } }
I20260923 20:12:25.150511 16644 raft_consensus.cc:348] T 691e285e2e31466e992aae1e9af38189 P d06de688bc57487a9a34608cd7c70b6a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d06de688bc57487a9a34608cd7c70b6a" member_type: VOTER last_known_addr { host: "127.1.170.131" port: 40407 } } peers { permanent_uuid: "53297953fdab4397adf94a117c453bc4" member_type: VOTER last_known_addr { host: "127.1.170.130" port: 37359 } } peers { permanent_uuid: "7f5b750bad6042cda73694b87b237180" member_type: VOTER last_known_addr { host: "127.1.170.129" port: 38865 } }
I20260923 20:12:25.150898 16644 raft_consensus.cc:374] T 691e285e2e31466e992aae1e9af38189 P d06de688bc57487a9a34608cd7c70b6a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:12:25.151005 16644 raft_consensus.cc:729] T 691e285e2e31466e992aae1e9af38189 P d06de688bc57487a9a34608cd7c70b6a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d06de688bc57487a9a34608cd7c70b6a, State: Initialized, Role: FOLLOWER
I20260923 20:12:25.151543 16644 consensus_queue.cc:260] T 691e285e2e31466e992aae1e9af38189 P d06de688bc57487a9a34608cd7c70b6a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d06de688bc57487a9a34608cd7c70b6a" member_type: VOTER last_known_addr { host: "127.1.170.131" port: 40407 } } peers { permanent_uuid: "53297953fdab4397adf94a117c453bc4" member_type: VOTER last_known_addr { host: "127.1.170.130" port: 37359 } } peers { permanent_uuid: "7f5b750bad6042cda73694b87b237180" member_type: VOTER last_known_addr { host: "127.1.170.129" port: 38865 } }
I20260923 20:12:25.152626 16626 heartbeater.cc:503] Master 127.1.170.190:43769 was elected leader, sending a full tablet report...
I20260923 20:12:25.153330 16644 ts_tablet_manager.cc:1428] T 691e285e2e31466e992aae1e9af38189 P d06de688bc57487a9a34608cd7c70b6a: Time spent starting tablet: real 0.012s user 0.004s sys 0.007s
I20260923 20:12:25.154563 16363 heartbeater.cc:503] Master 127.1.170.190:43769 was elected leader, sending a full tablet report...
I20260923 20:12:25.155745 16643 ts_tablet_manager.cc:1428] T 691e285e2e31466e992aae1e9af38189 P 7f5b750bad6042cda73694b87b237180: Time spent starting tablet: real 0.017s user 0.007s sys 0.009s
I20260923 20:12:25.156657 16495 heartbeater.cc:503] Master 127.1.170.190:43769 was elected leader, sending a full tablet report...
I20260923 20:12:25.157251 16642 ts_tablet_manager.cc:1428] T 691e285e2e31466e992aae1e9af38189 P 53297953fdab4397adf94a117c453bc4: Time spent starting tablet: real 0.020s user 0.009s sys 0.012s
I20260923 20:12:25.201143 16650 raft_consensus.cc:482] T 691e285e2e31466e992aae1e9af38189 P d06de688bc57487a9a34608cd7c70b6a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 20:12:25.201413 16650 raft_consensus.cc:504] T 691e285e2e31466e992aae1e9af38189 P d06de688bc57487a9a34608cd7c70b6a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d06de688bc57487a9a34608cd7c70b6a" member_type: VOTER last_known_addr { host: "127.1.170.131" port: 40407 } } peers { permanent_uuid: "53297953fdab4397adf94a117c453bc4" member_type: VOTER last_known_addr { host: "127.1.170.130" port: 37359 } } peers { permanent_uuid: "7f5b750bad6042cda73694b87b237180" member_type: VOTER last_known_addr { host: "127.1.170.129" port: 38865 } }
I20260923 20:12:25.203161 16650 leader_election.cc:288] T 691e285e2e31466e992aae1e9af38189 P d06de688bc57487a9a34608cd7c70b6a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 53297953fdab4397adf94a117c453bc4 (127.1.170.130:37359), 7f5b750bad6042cda73694b87b237180 (127.1.170.129:38865)
I20260923 20:12:25.209537 16628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:12:25.203592 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.129:38865 (local address 127.1.170.131:44375)
0923 20:12:25.204005 (+ 413us) negotiation.cc:107] Waiting for socket to connect
0923 20:12:25.204032 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0923 20:12:25.204226 (+ 194us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:12:25.204630 (+ 404us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:12:25.204637 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:12:25.204667 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:12:25.204869 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:12:25.204880 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:12:25.206100 (+ 1220us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:12:25.206106 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:12:25.207624 (+ 1518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:12:25.207635 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:12:25.208802 (+ 1167us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:12:25.208808 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:12:25.208994 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:12:25.209045 (+ 51us) client_negotiation.cc:769] Sending connection context
0923 20:12:25.209150 (+ 105us) client_negotiation.cc:240] Negotiation successful
0923 20:12:25.209279 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":223,"mutex_wait_us":56}
I20260923 20:12:25.209592 16641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:12:25.203609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.170.131:44375 (local address 127.1.170.129:38865)
0923 20:12:25.203813 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0923 20:12:25.203819 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:12:25.204142 (+ 323us) server_negotiation.cc:407] Connection header received
0923 20:12:25.204389 (+ 247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:12:25.204395 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:12:25.204486 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:12:25.204594 (+ 108us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:12:25.205038 (+ 444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:12:25.205944 (+ 906us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:12:25.207774 (+ 1830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:12:25.208646 (+ 872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:12:25.208760 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:12:25.208833 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0923 20:12:25.209190 (+ 357us) server_negotiation.cc:299] Negotiation successful
0923 20:12:25.209320 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260923 20:12:25.209906 16639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:12:25.203614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.170.131:37733 (local address 127.1.170.130:37359)
0923 20:12:25.203813 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0923 20:12:25.203819 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:12:25.204524 (+ 705us) server_negotiation.cc:407] Connection header received
0923 20:12:25.204704 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:12:25.204711 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:12:25.204780 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:12:25.204881 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:12:25.205481 (+ 600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:12:25.206327 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:12:25.208097 (+ 1770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:12:25.208959 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:12:25.209109 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:12:25.209185 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0923 20:12:25.209495 (+ 310us) server_negotiation.cc:299] Negotiation successful
0923 20:12:25.209663 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260923 20:12:25.209944 16651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:12:25.203672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.130:37359 (local address 127.1.170.131:37733)
0923 20:12:25.204401 (+ 729us) negotiation.cc:107] Waiting for socket to connect
0923 20:12:25.204417 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0923 20:12:25.204588 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:12:25.204900 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:12:25.204903 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:12:25.204921 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:12:25.205320 (+ 399us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:12:25.205333 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:12:25.206471 (+ 1138us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:12:25.206474 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:12:25.207947 (+ 1473us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:12:25.207960 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:12:25.209161 (+ 1201us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:12:25.209167 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:12:25.209333 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:12:25.209363 (+ 30us) client_negotiation.cc:769] Sending connection context
0923 20:12:25.209457 (+ 94us) client_negotiation.cc:240] Negotiation successful
0923 20:12:25.209587 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":596,"thread_start_us":422,"threads_started":1}
I20260923 20:12:25.210878 16450 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "691e285e2e31466e992aae1e9af38189" candidate_uuid: "d06de688bc57487a9a34608cd7c70b6a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "53297953fdab4397adf94a117c453bc4" is_pre_election: true
I20260923 20:12:25.210891 16318 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "691e285e2e31466e992aae1e9af38189" candidate_uuid: "d06de688bc57487a9a34608cd7c70b6a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f5b750bad6042cda73694b87b237180" is_pre_election: true
I20260923 20:12:25.211356 16450 raft_consensus.cc:2454] T 691e285e2e31466e992aae1e9af38189 P 53297953fdab4397adf94a117c453bc4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d06de688bc57487a9a34608cd7c70b6a in term 0.
I20260923 20:12:25.211416 16318 raft_consensus.cc:2454] T 691e285e2e31466e992aae1e9af38189 P 7f5b750bad6042cda73694b87b237180 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d06de688bc57487a9a34608cd7c70b6a in term 0.
I20260923 20:12:25.212105 16514 leader_election.cc:302] T 691e285e2e31466e992aae1e9af38189 P d06de688bc57487a9a34608cd7c70b6a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 53297953fdab4397adf94a117c453bc4, d06de688bc57487a9a34608cd7c70b6a; no voters:
I20260923 20:12:25.212515 16650 raft_consensus.cc:2781] T 691e285e2e31466e992aae1e9af38189 P d06de688bc57487a9a34608cd7c70b6a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 20:12:25.212657 16650 raft_consensus.cc:482] T 691e285e2e31466e992aae1e9af38189 P d06de688bc57487a9a34608cd7c70b6a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 20:12:25.212757 16650 raft_consensus.cc:3037] T 691e285e2e31466e992aae1e9af38189 P d06de688bc57487a9a34608cd7c70b6a [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:12:25.214982 16650 raft_consensus.cc:504] T 691e285e2e31466e992aae1e9af38189 P d06de688bc57487a9a34608cd7c70b6a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d06de688bc57487a9a34608cd7c70b6a" member_type: VOTER last_known_addr { host: "127.1.170.131" port: 40407 } } peers { permanent_uuid: "53297953fdab4397adf94a117c453bc4" member_type: VOTER last_known_addr { host: "127.1.170.130" port: 37359 } } peers { permanent_uuid: "7f5b750bad6042cda73694b87b237180" member_type: VOTER last_known_addr { host: "127.1.170.129" port: 38865 } }
I20260923 20:12:25.215613 16650 leader_election.cc:288] T 691e285e2e31466e992aae1e9af38189 P d06de688bc57487a9a34608cd7c70b6a [CANDIDATE]: Term 1 election: Requested vote from peers 53297953fdab4397adf94a117c453bc4 (127.1.170.130:37359), 7f5b750bad6042cda73694b87b237180 (127.1.170.129:38865)
I20260923 20:12:25.216046 16450 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "691e285e2e31466e992aae1e9af38189" candidate_uuid: "d06de688bc57487a9a34608cd7c70b6a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "53297953fdab4397adf94a117c453bc4"
I20260923 20:12:25.216136 16318 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "691e285e2e31466e992aae1e9af38189" candidate_uuid: "d06de688bc57487a9a34608cd7c70b6a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f5b750bad6042cda73694b87b237180"
I20260923 20:12:25.216269 16450 raft_consensus.cc:3037] T 691e285e2e31466e992aae1e9af38189 P 53297953fdab4397adf94a117c453bc4 [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:12:25.216336 16318 raft_consensus.cc:3037] T 691e285e2e31466e992aae1e9af38189 P 7f5b750bad6042cda73694b87b237180 [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:12:25.219520 16450 raft_consensus.cc:2454] T 691e285e2e31466e992aae1e9af38189 P 53297953fdab4397adf94a117c453bc4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d06de688bc57487a9a34608cd7c70b6a in term 1.
I20260923 20:12:25.219805 16318 raft_consensus.cc:2454] T 691e285e2e31466e992aae1e9af38189 P 7f5b750bad6042cda73694b87b237180 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d06de688bc57487a9a34608cd7c70b6a in term 1.
I20260923 20:12:25.219913 16514 leader_election.cc:302] T 691e285e2e31466e992aae1e9af38189 P d06de688bc57487a9a34608cd7c70b6a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 53297953fdab4397adf94a117c453bc4, d06de688bc57487a9a34608cd7c70b6a; no voters:
I20260923 20:12:25.220209 16650 raft_consensus.cc:2781] T 691e285e2e31466e992aae1e9af38189 P d06de688bc57487a9a34608cd7c70b6a [term 1 FOLLOWER]: Leader election won for term 1
I20260923 20:12:25.220659 16650 raft_consensus.cc:686] T 691e285e2e31466e992aae1e9af38189 P d06de688bc57487a9a34608cd7c70b6a [term 1 LEADER]: Becoming Leader. State: Replica: d06de688bc57487a9a34608cd7c70b6a, State: Running, Role: LEADER
I20260923 20:12:25.221170 16650 consensus_queue.cc:237] T 691e285e2e31466e992aae1e9af38189 P d06de688bc57487a9a34608cd7c70b6a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d06de688bc57487a9a34608cd7c70b6a" member_type: VOTER last_known_addr { host: "127.1.170.131" port: 40407 } } peers { permanent_uuid: "53297953fdab4397adf94a117c453bc4" member_type: VOTER last_known_addr { host: "127.1.170.130" port: 37359 } } peers { permanent_uuid: "7f5b750bad6042cda73694b87b237180" member_type: VOTER last_known_addr { host: "127.1.170.129" port: 38865 } }
I20260923 20:12:25.225550 16176 catalog_manager.cc:5495] T 691e285e2e31466e992aae1e9af38189 P d06de688bc57487a9a34608cd7c70b6a reported cstate change: term changed from 0 to 1, leader changed from <none> to d06de688bc57487a9a34608cd7c70b6a (127.1.170.131). New cstate: current_term: 1 leader_uuid: "d06de688bc57487a9a34608cd7c70b6a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d06de688bc57487a9a34608cd7c70b6a" member_type: VOTER last_known_addr { host: "127.1.170.131" port: 40407 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "53297953fdab4397adf94a117c453bc4" member_type: VOTER last_known_addr { host: "127.1.170.130" port: 37359 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7f5b750bad6042cda73694b87b237180" member_type: VOTER last_known_addr { host: "127.1.170.129" port: 38865 } health_report { overall_health: UNKNOWN } } }
I20260923 20:12:25.261116 16638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:12:25.255931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34726 (local address 127.1.170.131:40407)
0923 20:12:25.256111 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0923 20:12:25.256115 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 20:12:25.256131 (+ 16us) server_negotiation.cc:407] Connection header received
0923 20:12:25.256211 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:12:25.256214 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:12:25.256293 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:12:25.256411 (+ 118us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 20:12:25.256803 (+ 392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:12:25.257513 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:12:25.259037 (+ 1524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:12:25.259475 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:12:25.259582 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:12:25.260014 (+ 432us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 20:12:25.260525 (+ 511us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 20:12:25.260632 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0923 20:12:25.260726 (+ 94us) server_negotiation.cc:299] Negotiation successful
0923 20:12:25.260788 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260923 20:12:25.269091 16318 raft_consensus.cc:1261] T 691e285e2e31466e992aae1e9af38189 P 7f5b750bad6042cda73694b87b237180 [term 1 FOLLOWER]: Refusing update from remote peer d06de688bc57487a9a34608cd7c70b6a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 20:12:25.269133 16450 raft_consensus.cc:1261] T 691e285e2e31466e992aae1e9af38189 P 53297953fdab4397adf94a117c453bc4 [term 1 FOLLOWER]: Refusing update from remote peer d06de688bc57487a9a34608cd7c70b6a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 20:12:25.270049 16652 consensus_queue.cc:1035] T 691e285e2e31466e992aae1e9af38189 P d06de688bc57487a9a34608cd7c70b6a [LEADER]: Connected to new peer: Peer: permanent_uuid: "7f5b750bad6042cda73694b87b237180" member_type: VOTER last_known_addr { host: "127.1.170.129" port: 38865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 20:12:25.270442 16650 consensus_queue.cc:1035] T 691e285e2e31466e992aae1e9af38189 P d06de688bc57487a9a34608cd7c70b6a [LEADER]: Connected to new peer: Peer: permanent_uuid: "53297953fdab4397adf94a117c453bc4" member_type: VOTER last_known_addr { host: "127.1.170.130" port: 37359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260923 20:12:25.285008 16627 tablet.cc:2369] T 691e285e2e31466e992aae1e9af38189 P d06de688bc57487a9a34608cd7c70b6a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 20:12:25.286015 16655 mvcc.cc:205] Tried to move back new op lower bound from 7332636038212952064 to 7332636038029918208. Current Snapshot: MvccSnapshot[applied={T|T < 7332636038212952064}]
I20260923 20:12:25.287215 16657 mvcc.cc:205] Tried to move back new op lower bound from 7332636038212952064 to 7332636038029918208. Current Snapshot: MvccSnapshot[applied={T|T < 7332636038212952064}]
I20260923 20:12:25.302484 16176 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:41956:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 20:12:25.302772 16176 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:41956:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 20:12:25.306427 16176 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e910e26b0d9645f0bca3a3198101aa5a: Sending DeleteTablet for 3 replicas of tablet 691e285e2e31466e992aae1e9af38189
I20260923 20:12:25.307622 16430 tablet_service.cc:1514] Processing DeleteTablet for tablet 691e285e2e31466e992aae1e9af38189 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 20:12:25 UTC) from {username='kudu', principal='kudu/127.1.170.190@KRBTEST.COM'} at 127.0.0.1:49694
I20260923 20:12:25.307773 16298 tablet_service.cc:1514] Processing DeleteTablet for tablet 691e285e2e31466e992aae1e9af38189 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 20:12:25 UTC) from {username='kudu', principal='kudu/127.1.170.190@KRBTEST.COM'} at 127.0.0.1:41136
I20260923 20:12:25.307876 16561 tablet_service.cc:1514] Processing DeleteTablet for tablet 691e285e2e31466e992aae1e9af38189 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 20:12:25 UTC) from {username='kudu', principal='kudu/127.1.170.190@KRBTEST.COM'} at 127.0.0.1:34724
I20260923 20:12:25.308540 16665 tablet_replica.cc:330] T 691e285e2e31466e992aae1e9af38189 P 53297953fdab4397adf94a117c453bc4: stopping tablet replica
I20260923 20:12:25.308657 1706 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskqUuku6/build/asan/bin/kudu pid 16214
I20260923 20:12:25.308928 16665 raft_consensus.cc:2229] T 691e285e2e31466e992aae1e9af38189 P 53297953fdab4397adf94a117c453bc4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 20:12:25.309417 16665 raft_consensus.cc:2258] T 691e285e2e31466e992aae1e9af38189 P 53297953fdab4397adf94a117c453bc4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 20:12:25.311407 16665 ts_tablet_manager.cc:1905] T 691e285e2e31466e992aae1e9af38189 P 53297953fdab4397adf94a117c453bc4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 20:12:25.318041 16665 ts_tablet_manager.cc:1918] T 691e285e2e31466e992aae1e9af38189 P 53297953fdab4397adf94a117c453bc4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 20:12:25.318224 16665 log.cc:1196] T 691e285e2e31466e992aae1e9af38189 P 53297953fdab4397adf94a117c453bc4: Deleting WAL directory at /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194121577027-1706-0/minicluster-data/ts-1/wal/wals/691e285e2e31466e992aae1e9af38189
I20260923 20:12:25.318397 16667 tablet_replica.cc:330] T 691e285e2e31466e992aae1e9af38189 P 7f5b750bad6042cda73694b87b237180: stopping tablet replica
I20260923 20:12:25.318840 16665 ts_tablet_manager.cc:1939] T 691e285e2e31466e992aae1e9af38189 P 53297953fdab4397adf94a117c453bc4: Deleting consensus metadata
I20260923 20:12:25.319020 16667 raft_consensus.cc:2229] T 691e285e2e31466e992aae1e9af38189 P 7f5b750bad6042cda73694b87b237180 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 20:12:25.319612 16667 raft_consensus.cc:2258] T 691e285e2e31466e992aae1e9af38189 P 7f5b750bad6042cda73694b87b237180 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 20:12:25.321048 16161 catalog_manager.cc:4841] TS 53297953fdab4397adf94a117c453bc4 (127.1.170.130:37359): tablet 691e285e2e31466e992aae1e9af38189 (table test-table [id=06b14acadb5345d8a622475de12fcabe]) successfully deleted
I20260923 20:12:25.323621 16667 ts_tablet_manager.cc:1905] T 691e285e2e31466e992aae1e9af38189 P 7f5b750bad6042cda73694b87b237180: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 20:12:25.324873 16666 tablet_replica.cc:330] T 691e285e2e31466e992aae1e9af38189 P d06de688bc57487a9a34608cd7c70b6a: stopping tablet replica
I20260923 20:12:25.325314 16666 raft_consensus.cc:2229] T 691e285e2e31466e992aae1e9af38189 P d06de688bc57487a9a34608cd7c70b6a [term 1 LEADER]: Raft consensus shutting down.
I20260923 20:12:25.327670 16641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:12:25.309690 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41150 (local address 127.1.170.129:38865)
0923 20:12:25.309918 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0923 20:12:25.309926 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 20:12:25.309947 (+ 21us) server_negotiation.cc:407] Connection header received
0923 20:12:25.310044 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:12:25.310049 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:12:25.310139 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:12:25.310245 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:12:25.312611 (+ 2366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:12:25.314026 (+ 1415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:12:25.316616 (+ 2590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:12:25.317342 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:12:25.317479 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:12:25.321421 (+ 3942us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:12:25.321437 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:12:25.321455 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:12:25.321506 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:12:25.324554 (+ 3048us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:12:25.325452 (+ 898us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:12:25.325466 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:12:25.325469 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:12:25.325592 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:12:25.326069 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:12:25.326075 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:12:25.326078 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:12:25.326449 (+ 371us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:12:25.326767 (+ 318us) server_negotiation.cc:1035] Waiting for connection context
0923 20:12:25.327074 (+ 307us) server_negotiation.cc:299] Negotiation successful
0923 20:12:25.327306 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260923 20:12:25.328497 16358 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:12:25.328552 16666 raft_consensus.cc:2258] T 691e285e2e31466e992aae1e9af38189 P d06de688bc57487a9a34608cd7c70b6a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 20:12:25.331732 16666 ts_tablet_manager.cc:1905] T 691e285e2e31466e992aae1e9af38189 P d06de688bc57487a9a34608cd7c70b6a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 20:12:25.337268 16666 ts_tablet_manager.cc:1918] T 691e285e2e31466e992aae1e9af38189 P d06de688bc57487a9a34608cd7c70b6a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 20:12:25.337457 16666 log.cc:1196] T 691e285e2e31466e992aae1e9af38189 P d06de688bc57487a9a34608cd7c70b6a: Deleting WAL directory at /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194121577027-1706-0/minicluster-data/ts-2/wal/wals/691e285e2e31466e992aae1e9af38189
I20260923 20:12:25.338042 16666 ts_tablet_manager.cc:1939] T 691e285e2e31466e992aae1e9af38189 P d06de688bc57487a9a34608cd7c70b6a: Deleting consensus metadata
I20260923 20:12:25.339604 16163 catalog_manager.cc:4841] TS d06de688bc57487a9a34608cd7c70b6a (127.1.170.131:40407): tablet 691e285e2e31466e992aae1e9af38189 (table test-table [id=06b14acadb5345d8a622475de12fcabe]) successfully deleted
I20260923 20:12:25.458412 1706 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqUuku6/build/asan/bin/kudu with pid 16214
W20260923 20:12:25.473845 16163 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.1.170.129:38865 (error 108)
I20260923 20:12:25.474067 1706 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskqUuku6/build/asan/bin/kudu pid 16367
I20260923 20:12:25.477129 16640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:12:25.476638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.129:38865 (local address 127.0.0.1:41156)
0923 20:12:25.476831 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0923 20:12:25.476950 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.170.129:38865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
W20260923 20:12:25.477517 16163 catalog_manager.cc:4584] TS 7f5b750bad6042cda73694b87b237180 (127.1.170.129:38865): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 691e285e2e31466e992aae1e9af38189: Network error: Client connection negotiation failed: client connection to 127.1.170.129:38865: connect: Connection refused (error 111)
I20260923 20:12:25.488511 16639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:12:25.474933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49706 (local address 127.1.170.130:37359)
0923 20:12:25.475147 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0923 20:12:25.475154 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:12:25.475309 (+ 155us) server_negotiation.cc:407] Connection header received
0923 20:12:25.475535 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:12:25.475542 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:12:25.475630 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:12:25.475749 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:12:25.477225 (+ 1476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:12:25.478233 (+ 1008us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:12:25.479568 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:12:25.479970 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:12:25.480104 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:12:25.482809 (+ 2705us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:12:25.482822 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:12:25.482835 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:12:25.482867 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:12:25.485820 (+ 2953us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:12:25.486494 (+ 674us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:12:25.486507 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:12:25.486510 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:12:25.486629 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:12:25.486986 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:12:25.486992 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:12:25.486995 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:12:25.487378 (+ 383us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:12:25.487683 (+ 305us) server_negotiation.cc:1035] Waiting for connection context
0923 20:12:25.487966 (+ 283us) server_negotiation.cc:299] Negotiation successful
0923 20:12:25.488175 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260923 20:12:25.489166 16490 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:12:25.531386 16640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:12:25.530798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.129:38865 (local address 127.0.0.1:41166)
0923 20:12:25.531046 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0923 20:12:25.531182 (+ 136us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.170.129:38865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
I20260923 20:12:25.532585 16640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:12:25.532174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.129:38865 (local address 127.0.0.1:41178)
0923 20:12:25.532339 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0923 20:12:25.532452 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.170.129:38865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260923 20:12:25.594439 16640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:12:25.593011 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.129:38865 (local address 127.0.0.1:41192)
0923 20:12:25.594127 (+ 1116us) negotiation.cc:107] Waiting for socket to connect
0923 20:12:25.594254 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.170.129:38865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":937}
I20260923 20:12:25.595556 1706 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqUuku6/build/asan/bin/kudu with pid 16367
I20260923 20:12:25.597530 16640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:12:25.595570 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.129:38865 (local address 127.0.0.1:41204)
0923 20:12:25.597259 (+ 1689us) negotiation.cc:107] Waiting for socket to connect
0923 20:12:25.597367 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.170.129:38865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1560}
I20260923 20:12:25.620339 1706 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskqUuku6/build/asan/bin/kudu pid 16498
I20260923 20:12:25.641119 16638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:12:25.621169 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34736 (local address 127.1.170.131:40407)
0923 20:12:25.621336 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0923 20:12:25.621340 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 20:12:25.621359 (+ 19us) server_negotiation.cc:407] Connection header received
0923 20:12:25.621571 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:12:25.621575 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:12:25.621679 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:12:25.621770 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:12:25.623650 (+ 1880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:12:25.624865 (+ 1215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:12:25.627618 (+ 2753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:12:25.628346 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:12:25.628502 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:12:25.633081 (+ 4579us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:12:25.633100 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:12:25.633132 (+ 32us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:12:25.633191 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:12:25.637374 (+ 4183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:12:25.638487 (+ 1113us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:12:25.638497 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:12:25.638501 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:12:25.638739 (+ 238us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:12:25.639231 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:12:25.639236 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:12:25.639240 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:12:25.639751 (+ 511us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:12:25.640080 (+ 329us) server_negotiation.cc:1035] Waiting for connection context
0923 20:12:25.640410 (+ 330us) server_negotiation.cc:299] Negotiation successful
0923 20:12:25.640667 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260923 20:12:25.641997 16621 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:12:25.678179 16640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:12:25.677539 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.129:38865 (local address 127.0.0.1:41214)
0923 20:12:25.677789 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0923 20:12:25.677930 (+ 141us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.170.129:38865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92}
I20260923 20:12:25.679296 16640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:12:25.678852 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.129:38865 (local address 127.0.0.1:41220)
0923 20:12:25.679063 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0923 20:12:25.679161 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.170.129:38865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105}
I20260923 20:12:25.771070 1706 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqUuku6/build/asan/bin/kudu with pid 16498
I20260923 20:12:25.790899 1706 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskqUuku6/build/asan/bin/kudu pid 16144
I20260923 20:12:25.792279 16207 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:12:25.911031 16640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:12:25.909737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.129:38865 (local address 127.0.0.1:41222)
0923 20:12:25.910444 (+ 707us) negotiation.cc:107] Waiting for socket to connect
0923 20:12:25.910587 (+ 143us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.170.129:38865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":561}
I20260923 20:12:25.911374 1706 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqUuku6/build/asan/bin/kudu with pid 16144
I20260923 20:12:25.933013 1706 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqUuku6/build/asan/bin/kudu
/tmp/dist-test-taskqUuku6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194121577027-1706-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194121577027-1706-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194121577027-1706-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194121577027-1706-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.170.190:43769
--webserver_interface=127.1.170.190
--webserver_port=39057
--keytab_file=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194121577027-1706-0/krb5kdc/kudu_127.1.170.190.keytab
--principal=kudu/127.1.170.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.170.148:37943
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.170.190:43769
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194121577027-1706-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194121577027-1706-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194121577027-1706-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:12:26.284512 16673 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:12:26.284865 16673 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:12:26.285025 16673 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:12:26.292527 16673 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 20:12:26.292702 16673 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:12:26.292805 16673 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:12:26.292904 16673 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 20:12:26.292999 16673 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 20:12:26.303277 16673 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.170.148:37943
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194121577027-1706-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194121577027-1706-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.170.190:43769
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194121577027-1706-0/krb5kdc/kudu_127.1.170.190.keytab
--principal=kudu/127.1.170.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.170.190:43769
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194121577027-1706-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.170.190
--webserver_port=39057
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194121577027-1706-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:12:26.304663 16673 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:12:26.305977 16673 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:12:26.321010 16678 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:12:26.321178 16681 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:12:26.320935 16679 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:12:26.322666 16673 server_base.cc:1034] running on GCE node
I20260923 20:12:26.323915 16673 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:12:26.325965 16673 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:12:26.327353 16673 hybrid_clock.cc:648] HybridClock initialized: now 1790194346327284 us; error 50 us; skew 500 ppm
I20260923 20:12:26.327977 16673 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:12:26 dist-test-slave-th4g krb5kdc[15592](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194346, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.170.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:12:26.332546 16673 init.cc:377] Logged in from keytab as kudu/127.1.170.190@KRBTEST.COM (short username kudu)
I20260923 20:12:26.334512 16673 webserver.cc:460] Webserver started at http://127.1.170.190:39057/ using document root <none> and password file <none>
I20260923 20:12:26.335263 16673 fs_manager.cc:359] Metadata directory not provided
I20260923 20:12:26.335392 16673 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:12:26.343973 16673 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.002s sys 0.002s
I20260923 20:12:26.347708 16688 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:12:26.349632 16673 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260923 20:12:26.349898 16673 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194121577027-1706-0/minicluster-data/master-0/data,/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194121577027-1706-0/minicluster-data/master-0/wal
uuid: "e910e26b0d9645f0bca3a3198101aa5a"
format_stamp: "Formatted at 2026-09-23 20:12:20 on dist-test-slave-th4g"
server_key: "0322a97aa01b78f36a69b9b99ac35141"
server_key_iv: "469c0c9a2ee402c69b62f6ea74f31bac"
server_key_version: "encryptionkey@0"
I20260923 20:12:26.351238 16673 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194121577027-1706-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194121577027-1706-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194121577027-1706-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:12:26.387250 16673 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:12:26.392216 16673 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:12:26.392526 16673 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:12:26.412871 16673 rpc_server.cc:307] RPC server started. Bound to: 127.1.170.190:43769
I20260923 20:12:26.412901 16739 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.170.190:43769 every 8 connection(s)
I20260923 20:12:26.414420 16673 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194121577027-1706-0/minicluster-data/master-0/data/info.pb
I20260923 20:12:26.418150 1706 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqUuku6/build/asan/bin/kudu as pid 16673
I20260923 20:12:26.418988 1706 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqUuku6/build/asan/bin/kudu
/tmp/dist-test-taskqUuku6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194121577027-1706-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194121577027-1706-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194121577027-1706-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194121577027-1706-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.170.129:38865
--local_ip_for_outbound_sockets=127.1.170.129
--tserver_master_addrs=127.1.170.190:43769
--webserver_port=42049
--webserver_interface=127.1.170.129
--keytab_file=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194121577027-1706-0/krb5kdc/kudu_127.1.170.129.keytab
--principal=kudu/127.1.170.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.170.148:37943
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194121577027-1706-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194121577027-1706-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194121577027-1706-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260923 20:12:26.422029 16740 sys_catalog.cc:263] Verifying existing consensus state
I20260923 20:12:26.425451 16740 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e910e26b0d9645f0bca3a3198101aa5a: Bootstrap starting.
I20260923 20:12:26.446658 16740 log.cc:824] T 00000000000000000000000000000000 P e910e26b0d9645f0bca3a3198101aa5a: Log is configured to *not* fsync() on all Append() calls
I20260923 20:12:26.463603 16740 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e910e26b0d9645f0bca3a3198101aa5a: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260923 20:12:26.464226 16740 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e910e26b0d9645f0bca3a3198101aa5a: Bootstrap complete.
I20260923 20:12:26.475409 16740 raft_consensus.cc:348] T 00000000000000000000000000000000 P e910e26b0d9645f0bca3a3198101aa5a [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e910e26b0d9645f0bca3a3198101aa5a" member_type: VOTER last_known_addr { host: "127.1.170.190" port: 43769 } }
I20260923 20:12:26.476601 16740 raft_consensus.cc:729] T 00000000000000000000000000000000 P e910e26b0d9645f0bca3a3198101aa5a [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: e910e26b0d9645f0bca3a3198101aa5a, State: Initialized, Role: FOLLOWER
I20260923 20:12:26.477269 16740 consensus_queue.cc:260] T 00000000000000000000000000000000 P e910e26b0d9645f0bca3a3198101aa5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e910e26b0d9645f0bca3a3198101aa5a" member_type: VOTER last_known_addr { host: "127.1.170.190" port: 43769 } }
I20260923 20:12:26.477527 16740 raft_consensus.cc:388] T 00000000000000000000000000000000 P e910e26b0d9645f0bca3a3198101aa5a [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 20:12:26.477658 16740 raft_consensus.cc:482] T 00000000000000000000000000000000 P e910e26b0d9645f0bca3a3198101aa5a [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 20:12:26.477846 16740 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e910e26b0d9645f0bca3a3198101aa5a [term 2 FOLLOWER]: Advancing to term 3
I20260923 20:12:26.481904 16740 raft_consensus.cc:504] T 00000000000000000000000000000000 P e910e26b0d9645f0bca3a3198101aa5a [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e910e26b0d9645f0bca3a3198101aa5a" member_type: VOTER last_known_addr { host: "127.1.170.190" port: 43769 } }
I20260923 20:12:26.482496 16740 leader_election.cc:302] T 00000000000000000000000000000000 P e910e26b0d9645f0bca3a3198101aa5a [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e910e26b0d9645f0bca3a3198101aa5a; no voters:
I20260923 20:12:26.483193 16740 leader_election.cc:288] T 00000000000000000000000000000000 P e910e26b0d9645f0bca3a3198101aa5a [CANDIDATE]: Term 3 election: Requested vote from peers
I20260923 20:12:26.483335 16744 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e910e26b0d9645f0bca3a3198101aa5a [term 3 FOLLOWER]: Leader election won for term 3
I20260923 20:12:26.484539 16744 raft_consensus.cc:686] T 00000000000000000000000000000000 P e910e26b0d9645f0bca3a3198101aa5a [term 3 LEADER]: Becoming Leader. State: Replica: e910e26b0d9645f0bca3a3198101aa5a, State: Running, Role: LEADER
I20260923 20:12:26.485157 16744 consensus_queue.cc:237] T 00000000000000000000000000000000 P e910e26b0d9645f0bca3a3198101aa5a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e910e26b0d9645f0bca3a3198101aa5a" member_type: VOTER last_known_addr { host: "127.1.170.190" port: 43769 } }
I20260923 20:12:26.485546 16740 sys_catalog.cc:564] T 00000000000000000000000000000000 P e910e26b0d9645f0bca3a3198101aa5a [sys.catalog]: configured and running, proceeding with master startup.
I20260923 20:12:26.488281 16746 sys_catalog.cc:455] T 00000000000000000000000000000000 P e910e26b0d9645f0bca3a3198101aa5a [sys.catalog]: SysCatalogTable state changed. Reason: New leader e910e26b0d9645f0bca3a3198101aa5a. Latest consensus state: current_term: 3 leader_uuid: "e910e26b0d9645f0bca3a3198101aa5a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e910e26b0d9645f0bca3a3198101aa5a" member_type: VOTER last_known_addr { host: "127.1.170.190" port: 43769 } } }
I20260923 20:12:26.488757 16746 sys_catalog.cc:458] T 00000000000000000000000000000000 P e910e26b0d9645f0bca3a3198101aa5a [sys.catalog]: This master's current role is: LEADER
I20260923 20:12:26.489346 16745 sys_catalog.cc:455] T 00000000000000000000000000000000 P e910e26b0d9645f0bca3a3198101aa5a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "e910e26b0d9645f0bca3a3198101aa5a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e910e26b0d9645f0bca3a3198101aa5a" member_type: VOTER last_known_addr { host: "127.1.170.190" port: 43769 } } }
I20260923 20:12:26.489650 16745 sys_catalog.cc:458] T 00000000000000000000000000000000 P e910e26b0d9645f0bca3a3198101aa5a [sys.catalog]: This master's current role is: LEADER
I20260923 20:12:26.501196 16750 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 20:12:26.511703 16750 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 20:12:26.513464 16750 catalog_manager.cc:1259] Loaded cluster ID: 26a0561cd2ff469cb55904b448b78043
I20260923 20:12:26.513582 16750 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260923 20:12:26.515972 16750 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f605b5ae9ad google::LogMessage::Fail() at ??:0
@ 0x7f605b5b188a google::LogMessage::SendToLog() at ??:0
@ 0x7f605b5ae4de google::LogMessage::Flush() at ??:0
@ 0x7f605b5afa19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f6070fe1512 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7f6070fde976 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f6071066610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7f6071066439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f605e1fa345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f605e1fd91c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7f605e1fd799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f605e1d8eaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7f60614b26db start_thread at ??:0
@ 0x7f605a44861f clone at ??:0
W20260923 20:12:26.842270 16742 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:12:26.842573 16742 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:12:26.842711 16742 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:12:26.850240 16742 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:12:26.850368 16742 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:12:26.850565 16742 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.170.129
I20260923 20:12:26.860883 16742 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.170.148:37943
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194121577027-1706-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194121577027-1706-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194121577027-1706-0/krb5kdc/kudu_127.1.170.129.keytab
--principal=kudu/127.1.170.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.170.129:38865
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194121577027-1706-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.170.129
--webserver_port=42049
--tserver_master_addrs=127.1.170.190:43769
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.170.129
--log_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194121577027-1706-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:12:26.862473 16742 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:12:26.863977 16742 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:12:26.879103 16784 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:12:26.879149 16782 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:12:26.879716 16781 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:12:26.880235 16742 server_base.cc:1034] running on GCE node
I20260923 20:12:26.880942 16742 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:12:26.882155 16742 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:12:26.883419 16742 hybrid_clock.cc:648] HybridClock initialized: now 1790194346883342 us; error 76 us; skew 500 ppm
I20260923 20:12:26.883800 16742 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:12:26 dist-test-slave-th4g krb5kdc[15592](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194346, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.170.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:12:26.888422 16742 init.cc:377] Logged in from keytab as kudu/127.1.170.129@KRBTEST.COM (short username kudu)
I20260923 20:12:26.890194 16742 webserver.cc:460] Webserver started at http://127.1.170.129:42049/ using document root <none> and password file <none>
I20260923 20:12:26.891052 16742 fs_manager.cc:359] Metadata directory not provided
I20260923 20:12:26.891207 16742 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:12:26.897635 16742 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.003s
I20260923 20:12:26.901454 16791 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:12:26.903668 16742 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260923 20:12:26.903913 16742 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194121577027-1706-0/minicluster-data/ts-0/data,/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194121577027-1706-0/minicluster-data/ts-0/wal
uuid: "7f5b750bad6042cda73694b87b237180"
format_stamp: "Formatted at 2026-09-23 20:12:21 on dist-test-slave-th4g"
server_key: "0ef88f7572d94c5b107314394af66393"
server_key_iv: "4fc80b00dd9f302d00610d10530c0ad8"
server_key_version: "encryptionkey@0"
I20260923 20:12:26.905028 16742 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194121577027-1706-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194121577027-1706-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194121577027-1706-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:12:26.921744 16742 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:12:26.927194 16742 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:12:26.927562 16742 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:12:26.928678 16742 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 20:12:26.930924 16798 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
W20260923 20:12:26.937474 16798 ts_tablet_manager.cc:1852] T 691e285e2e31466e992aae1e9af38189 P 7f5b750bad6042cda73694b87b237180: Tablet Manager startup: Rolling forward tablet deletion of type TABLET_DATA_DELETED
I20260923 20:12:26.938618 16798 ts_tablet_manager.cc:1905] T 691e285e2e31466e992aae1e9af38189 P 7f5b750bad6042cda73694b87b237180: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 20:12:26.946749 16798 ts_tablet_manager.cc:1918] T 691e285e2e31466e992aae1e9af38189 P 7f5b750bad6042cda73694b87b237180: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 20:12:26.946960 16798 log.cc:1196] T 691e285e2e31466e992aae1e9af38189 P 7f5b750bad6042cda73694b87b237180: Deleting WAL directory at /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194121577027-1706-0/minicluster-data/ts-0/wal/wals/691e285e2e31466e992aae1e9af38189
I20260923 20:12:26.947537 16798 ts_tablet_manager.cc:1939] T 691e285e2e31466e992aae1e9af38189 P 7f5b750bad6042cda73694b87b237180: Deleting consensus metadata
I20260923 20:12:26.948040 16742 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 0 live tablets)
I20260923 20:12:26.948272 16742 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.018s user 0.000s sys 0.001s
I20260923 20:12:26.948475 16742 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 20:12:26.948583 16742 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 20:12:26.986322 16742 rpc_server.cc:307] RPC server started. Bound to: 127.1.170.129:38865
I20260923 20:12:26.986452 16904 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.170.129:38865 every 8 connection(s)
I20260923 20:12:26.987684 16742 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194121577027-1706-0/minicluster-data/ts-0/data/info.pb
I20260923 20:12:26.990590 1706 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqUuku6/build/asan/bin/kudu as pid 16742
I20260923 20:12:26.991389 1706 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqUuku6/build/asan/bin/kudu
/tmp/dist-test-taskqUuku6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194121577027-1706-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194121577027-1706-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194121577027-1706-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194121577027-1706-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.170.130:37359
--local_ip_for_outbound_sockets=127.1.170.130
--tserver_master_addrs=127.1.170.190:43769
--webserver_port=41715
--webserver_interface=127.1.170.130
--keytab_file=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194121577027-1706-0/krb5kdc/kudu_127.1.170.130.keytab
--principal=kudu/127.1.170.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.170.148:37943
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194121577027-1706-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194121577027-1706-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194121577027-1706-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260923 20:12:26.993546 16907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:12:26.991168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.190:43769 (local address 127.1.170.129:59343)
0923 20:12:26.992395 (+ 1227us) negotiation.cc:107] Waiting for socket to connect
0923 20:12:26.992661 (+ 266us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.170.190:43769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":946,"thread_start_us":639,"threads_started":1}
W20260923 20:12:26.994203 16905 heartbeater.cc:644] Failed to heartbeat to 127.1.170.190:43769 (0 consecutive failures): Network error: Failed to ping master at 127.1.170.190:43769: Client connection negotiation failed: client connection to 127.1.170.190:43769: connect: Connection refused (error 111)
I20260923 20:12:26.995220 16907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:12:26.994812 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.190:43769 (local address 127.1.170.129:33181)
0923 20:12:26.995002 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0923 20:12:26.995078 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.170.190:43769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
I20260923 20:12:26.996313 16907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:12:26.995917 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.190:43769 (local address 127.1.170.129:49021)
0923 20:12:26.996072 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0923 20:12:26.996160 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.170.190:43769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
W20260923 20:12:26.996556 16905 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260923 20:12:27.335891 16908 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:12:27.336205 16908 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:12:27.336302 16908 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:12:27.343623 16908 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:12:27.343730 16908 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:12:27.343843 16908 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.170.130
I20260923 20:12:27.353679 16908 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.170.148:37943
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194121577027-1706-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194121577027-1706-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194121577027-1706-0/krb5kdc/kudu_127.1.170.130.keytab
--principal=kudu/127.1.170.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.170.130:37359
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194121577027-1706-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.170.130
--webserver_port=41715
--tserver_master_addrs=127.1.170.190:43769
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.170.130
--log_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194121577027-1706-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:12:27.355286 16908 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:12:27.356670 16908 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:12:27.370074 16913 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:12:27.370208 16916 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:12:27.370236 16914 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:12:27.371347 16908 server_base.cc:1034] running on GCE node
I20260923 20:12:27.372007 16908 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:12:27.373000 16908 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:12:27.374228 16908 hybrid_clock.cc:648] HybridClock initialized: now 1790194347374181 us; error 40 us; skew 500 ppm
I20260923 20:12:27.374557 16908 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:12:27 dist-test-slave-th4g krb5kdc[15592](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194347, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.170.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:12:27.377919 16908 init.cc:377] Logged in from keytab as kudu/127.1.170.130@KRBTEST.COM (short username kudu)
I20260923 20:12:27.379379 16908 webserver.cc:460] Webserver started at http://127.1.170.130:41715/ using document root <none> and password file <none>
I20260923 20:12:27.380076 16908 fs_manager.cc:359] Metadata directory not provided
I20260923 20:12:27.380216 16908 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:12:27.385670 16908 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.001s
I20260923 20:12:27.388592 16923 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:12:27.390153 16908 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 20:12:27.390340 16908 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194121577027-1706-0/minicluster-data/ts-1/data,/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194121577027-1706-0/minicluster-data/ts-1/wal
uuid: "53297953fdab4397adf94a117c453bc4"
format_stamp: "Formatted at 2026-09-23 20:12:21 on dist-test-slave-th4g"
server_key: "3f95013455082ede0c61607aa4f14e3e"
server_key_iv: "546d26b1ce46b518a3cf9e42439abc2f"
server_key_version: "encryptionkey@0"
I20260923 20:12:27.391386 16908 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194121577027-1706-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194121577027-1706-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194121577027-1706-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:12:27.412464 16908 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:12:27.417024 16908 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:12:27.417330 16908 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:12:27.418417 16908 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 20:12:27.419760 16908 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 20:12:27.419890 16908 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 20:12:27.420015 16908 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 20:12:27.420099 16908 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 20:12:27.454258 16908 rpc_server.cc:307] RPC server started. Bound to: 127.1.170.130:37359
I20260923 20:12:27.454324 17035 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.170.130:37359 every 8 connection(s)
I20260923 20:12:27.455717 16908 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194121577027-1706-0/minicluster-data/ts-1/data/info.pb
I20260923 20:12:27.459044 1706 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqUuku6/build/asan/bin/kudu as pid 16908
I20260923 20:12:27.459875 1706 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqUuku6/build/asan/bin/kudu
/tmp/dist-test-taskqUuku6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194121577027-1706-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194121577027-1706-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194121577027-1706-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194121577027-1706-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.170.131:40407
--local_ip_for_outbound_sockets=127.1.170.131
--tserver_master_addrs=127.1.170.190:43769
--webserver_port=34681
--webserver_interface=127.1.170.131
--keytab_file=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194121577027-1706-0/krb5kdc/kudu_127.1.170.131.keytab
--principal=kudu/127.1.170.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.170.148:37943
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194121577027-1706-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194121577027-1706-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194121577027-1706-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260923 20:12:27.460798 17038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:12:27.458543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.190:43769 (local address 127.1.170.130:53655)
0923 20:12:27.459376 (+ 833us) negotiation.cc:107] Waiting for socket to connect
0923 20:12:27.459689 (+ 313us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.170.190:43769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":521,"thread_start_us":303,"threads_started":1}
W20260923 20:12:27.461611 17036 heartbeater.cc:644] Failed to heartbeat to 127.1.170.190:43769 (0 consecutive failures): Network error: Failed to ping master at 127.1.170.190:43769: Client connection negotiation failed: client connection to 127.1.170.190:43769: connect: Connection refused (error 111)
I20260923 20:12:27.462591 17038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:12:27.462140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.190:43769 (local address 127.1.170.130:34061)
0923 20:12:27.462332 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0923 20:12:27.462433 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.170.190:43769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260923 20:12:27.463567 17038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:12:27.463156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.190:43769 (local address 127.1.170.130:42133)
0923 20:12:27.463319 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0923 20:12:27.463417 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.170.190:43769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260923 20:12:27.463829 17036 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260923 20:12:27.794020 17039 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:12:27.794322 17039 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:12:27.794415 17039 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:12:27.801731 17039 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:12:27.801842 17039 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:12:27.802078 17039 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.170.131
I20260923 20:12:27.812129 17039 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.170.148:37943
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194121577027-1706-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194121577027-1706-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194121577027-1706-0/krb5kdc/kudu_127.1.170.131.keytab
--principal=kudu/127.1.170.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.170.131:40407
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194121577027-1706-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.170.131
--webserver_port=34681
--tserver_master_addrs=127.1.170.190:43769
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.170.131
--log_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194121577027-1706-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:12:27.813482 17039 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:12:27.814829 17039 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:12:27.827888 17044 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:12:27.827797 17047 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:12:27.827893 17045 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:12:27.829214 17039 server_base.cc:1034] running on GCE node
I20260923 20:12:27.829833 17039 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:12:27.830881 17039 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:12:27.832275 17039 hybrid_clock.cc:648] HybridClock initialized: now 1790194347832208 us; error 57 us; skew 500 ppm
I20260923 20:12:27.832628 17039 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:12:27 dist-test-slave-th4g krb5kdc[15592](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194347, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.170.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:12:27.836102 17039 init.cc:377] Logged in from keytab as kudu/127.1.170.131@KRBTEST.COM (short username kudu)
I20260923 20:12:27.837697 17039 webserver.cc:460] Webserver started at http://127.1.170.131:34681/ using document root <none> and password file <none>
I20260923 20:12:27.838459 17039 fs_manager.cc:359] Metadata directory not provided
I20260923 20:12:27.838604 17039 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:12:27.843976 17039 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.003s
I20260923 20:12:27.846799 17054 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:12:27.848114 17039 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 20:12:27.848285 17039 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194121577027-1706-0/minicluster-data/ts-2/data,/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194121577027-1706-0/minicluster-data/ts-2/wal
uuid: "d06de688bc57487a9a34608cd7c70b6a"
format_stamp: "Formatted at 2026-09-23 20:12:22 on dist-test-slave-th4g"
server_key: "6f84fe378bd8c0618a4fe5ebbe74fe2e"
server_key_iv: "344643454d475f00f5172ce964fc04f8"
server_key_version: "encryptionkey@0"
I20260923 20:12:27.849249 17039 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194121577027-1706-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194121577027-1706-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194121577027-1706-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:12:27.866868 17039 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:12:27.870957 17039 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:12:27.871229 17039 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:12:27.872197 17039 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 20:12:27.873517 17039 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 20:12:27.873660 17039 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 20:12:27.873781 17039 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 20:12:27.873919 17039 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 20:12:27.907191 17039 rpc_server.cc:307] RPC server started. Bound to: 127.1.170.131:40407
I20260923 20:12:27.907276 17166 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.170.131:40407 every 8 connection(s)
I20260923 20:12:27.908593 17039 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194121577027-1706-0/minicluster-data/ts-2/data/info.pb
I20260923 20:12:27.913311 17169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:12:27.911614 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.190:43769 (local address 127.1.170.131:33311)
0923 20:12:27.912289 (+ 675us) negotiation.cc:107] Waiting for socket to connect
0923 20:12:27.912508 (+ 219us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.170.190:43769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":439,"thread_start_us":260,"threads_started":1}
W20260923 20:12:27.914040 17167 heartbeater.cc:644] Failed to heartbeat to 127.1.170.190:43769 (0 consecutive failures): Network error: Failed to ping master at 127.1.170.190:43769: Client connection negotiation failed: client connection to 127.1.170.190:43769: connect: Connection refused (error 111)
I20260923 20:12:27.914803 17169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:12:27.914504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.190:43769 (local address 127.1.170.131:42109)
0923 20:12:27.914640 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0923 20:12:27.914704 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.170.190:43769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260923 20:12:27.915653 17169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:12:27.915352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.190:43769 (local address 127.1.170.131:42395)
0923 20:12:27.915466 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0923 20:12:27.915535 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.170.190:43769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260923 20:12:27.915901 17167 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260923 20:12:27.916782 1706 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqUuku6/build/asan/bin/kudu as pid 17039
I20260923 20:12:27.997831 17171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:12:27.997050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.190:43769 (local address 127.1.170.129:36795)
0923 20:12:27.997511 (+ 461us) negotiation.cc:107] Waiting for socket to connect
0923 20:12:27.997599 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.170.190:43769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":351,"thread_start_us":223,"threads_started":1}
I20260923 20:12:28.465371 17172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:12:28.464536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.190:43769 (local address 127.1.170.130:55739)
0923 20:12:28.465001 (+ 465us) negotiation.cc:107] Waiting for socket to connect
0923 20:12:28.465141 (+ 140us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.170.190:43769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":356,"thread_start_us":228,"threads_started":1}
I20260923 20:12:28.917225 17173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:12:28.916456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.190:43769 (local address 127.1.170.131:37283)
0923 20:12:28.916888 (+ 432us) negotiation.cc:107] Waiting for socket to connect
0923 20:12:28.916995 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.170.190:43769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":335,"thread_start_us":212,"threads_started":1}
I20260923 20:12:28.999421 17174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:12:28.998741 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.190:43769 (local address 127.1.170.129:50077)
0923 20:12:28.999183 (+ 442us) negotiation.cc:107] Waiting for socket to connect
0923 20:12:28.999266 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.170.190:43769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":339,"thread_start_us":233,"threads_started":1}
I20260923 20:12:29.467343 17175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:12:29.466449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.190:43769 (local address 127.1.170.130:43741)
0923 20:12:29.467089 (+ 640us) negotiation.cc:107] Waiting for socket to connect
0923 20:12:29.467179 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.170.190:43769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":501,"thread_start_us":344,"threads_started":1}
I20260923 20:12:29.919333 17176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:12:29.918473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.190:43769 (local address 127.1.170.131:39195)
0923 20:12:29.919060 (+ 587us) negotiation.cc:107] Waiting for socket to connect
0923 20:12:29.919159 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.170.190:43769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":467,"thread_start_us":321,"threads_started":1}
I20260923 20:12:30.001322 17177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:12:30.000509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.190:43769 (local address 127.1.170.129:55457)
0923 20:12:30.000996 (+ 487us) negotiation.cc:107] Waiting for socket to connect
0923 20:12:30.001095 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.170.190:43769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":388,"thread_start_us":256,"threads_started":1}
I20260923 20:12:30.469377 17178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:12:30.468588 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.190:43769 (local address 127.1.170.130:55419)
0923 20:12:30.469099 (+ 511us) negotiation.cc:107] Waiting for socket to connect
0923 20:12:30.469200 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.170.190:43769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":393,"thread_start_us":263,"threads_started":1}
I20260923 20:12:30.921408 17179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:12:30.920580 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.190:43769 (local address 127.1.170.131:57223)
0923 20:12:30.921130 (+ 550us) negotiation.cc:107] Waiting for socket to connect
0923 20:12:30.921237 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.170.190:43769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":433,"thread_start_us":318,"threads_started":1}
I20260923 20:12:31.003459 17180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:12:31.002466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.190:43769 (local address 127.1.170.129:38867)
0923 20:12:31.003198 (+ 732us) negotiation.cc:107] Waiting for socket to connect
0923 20:12:31.003286 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.170.190:43769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":624,"thread_start_us":465,"threads_started":1}
I20260923 20:12:31.471385 17181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:12:31.470538 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.190:43769 (local address 127.1.170.130:44881)
0923 20:12:31.471095 (+ 557us) negotiation.cc:107] Waiting for socket to connect
0923 20:12:31.471192 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.170.190:43769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":453,"thread_start_us":353,"threads_started":1}
I20260923 20:12:31.923295 17182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:12:31.922513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.190:43769 (local address 127.1.170.131:58721)
0923 20:12:31.922967 (+ 454us) negotiation.cc:107] Waiting for socket to connect
0923 20:12:31.923052 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.170.190:43769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":343,"thread_start_us":224,"threads_started":1}
I20260923 20:12:32.005928 17183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:12:32.004464 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.190:43769 (local address 127.1.170.129:47025)
0923 20:12:32.005039 (+ 575us) negotiation.cc:107] Waiting for socket to connect
0923 20:12:32.005668 (+ 629us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.170.190:43769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":430,"thread_start_us":297,"threads_started":1}
I20260923 20:12:32.473266 17184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:12:32.472464 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.190:43769 (local address 127.1.170.130:60431)
0923 20:12:32.472995 (+ 531us) negotiation.cc:107] Waiting for socket to connect
0923 20:12:32.473097 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.170.190:43769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":413,"thread_start_us":280,"threads_started":1}
I20260923 20:12:32.925209 17185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:12:32.924415 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.190:43769 (local address 127.1.170.131:56855)
0923 20:12:32.924931 (+ 516us) negotiation.cc:107] Waiting for socket to connect
0923 20:12:32.925033 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.170.190:43769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":413,"thread_start_us":280,"threads_started":1}
I20260923 20:12:33.007604 17186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:12:33.006852 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.190:43769 (local address 127.1.170.129:53367)
0923 20:12:33.007341 (+ 489us) negotiation.cc:107] Waiting for socket to connect
0923 20:12:33.007431 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.170.190:43769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":390,"thread_start_us":274,"threads_started":1}
I20260923 20:12:33.475183 17187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:12:33.474417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.190:43769 (local address 127.1.170.130:34417)
0923 20:12:33.474900 (+ 483us) negotiation.cc:107] Waiting for socket to connect
0923 20:12:33.474997 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.170.190:43769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":378,"thread_start_us":256,"threads_started":1}
I20260923 20:12:33.927054 17188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:12:33.926266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.190:43769 (local address 127.1.170.131:58645)
0923 20:12:33.926769 (+ 503us) negotiation.cc:107] Waiting for socket to connect
0923 20:12:33.926869 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.170.190:43769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":392,"thread_start_us":259,"threads_started":1}
I20260923 20:12:34.009725 17189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:12:34.008790 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.190:43769 (local address 127.1.170.129:52015)
0923 20:12:34.009416 (+ 626us) negotiation.cc:107] Waiting for socket to connect
0923 20:12:34.009503 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.170.190:43769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":509,"thread_start_us":353,"threads_started":1}
I20260923 20:12:34.477615 17190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:12:34.476587 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.190:43769 (local address 127.1.170.130:42357)
0923 20:12:34.477294 (+ 707us) negotiation.cc:107] Waiting for socket to connect
0923 20:12:34.477391 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.170.190:43769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":587,"thread_start_us":266,"threads_started":1}
I20260923 20:12:34.929426 17191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:12:34.928389 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.190:43769 (local address 127.1.170.131:48049)
0923 20:12:34.929022 (+ 633us) negotiation.cc:107] Waiting for socket to connect
0923 20:12:34.929178 (+ 156us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.170.190:43769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":499,"thread_start_us":278,"threads_started":1}
I20260923 20:12:35.011588 17192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:12:35.010825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.190:43769 (local address 127.1.170.129:40919)
0923 20:12:35.011326 (+ 501us) negotiation.cc:107] Waiting for socket to connect
0923 20:12:35.011410 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.170.190:43769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":397,"thread_start_us":283,"threads_started":1}
I20260923 20:12:35.479825 17193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:12:35.478906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.190:43769 (local address 127.1.170.130:56969)
0923 20:12:35.479451 (+ 545us) negotiation.cc:107] Waiting for socket to connect
0923 20:12:35.479599 (+ 148us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.170.190:43769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":432,"thread_start_us":299,"threads_started":1}
I20260923 20:12:35.931628 17194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:12:35.930803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.190:43769 (local address 127.1.170.131:59859)
0923 20:12:35.931306 (+ 503us) negotiation.cc:107] Waiting for socket to connect
0923 20:12:35.931396 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.170.190:43769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":397,"thread_start_us":273,"threads_started":1}
I20260923 20:12:36.013543 17195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:12:36.012666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.190:43769 (local address 127.1.170.129:58189)
0923 20:12:36.013243 (+ 577us) negotiation.cc:107] Waiting for socket to connect
0923 20:12:36.013347 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.170.190:43769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":453,"thread_start_us":314,"threads_started":1}
I20260923 20:12:36.482430 17196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:12:36.481296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.190:43769 (local address 127.1.170.130:60733)
0923 20:12:36.482150 (+ 854us) negotiation.cc:107] Waiting for socket to connect
0923 20:12:36.482245 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.170.190:43769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":716,"thread_start_us":584,"threads_started":1}
I20260923 20:12:36.933717 17197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:12:36.932784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.190:43769 (local address 127.1.170.131:36027)
0923 20:12:36.933425 (+ 641us) negotiation.cc:107] Waiting for socket to connect
0923 20:12:36.933515 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.170.190:43769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":520,"thread_start_us":387,"threads_started":1}
I20260923 20:12:37.015678 17198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:12:37.014835 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.190:43769 (local address 127.1.170.129:39055)
0923 20:12:37.015410 (+ 575us) negotiation.cc:107] Waiting for socket to connect
0923 20:12:37.015496 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.170.190:43769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":464,"thread_start_us":359,"threads_started":1}
I20260923 20:12:37.484385 17199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:12:37.483519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.190:43769 (local address 127.1.170.130:56693)
0923 20:12:37.484128 (+ 609us) negotiation.cc:107] Waiting for socket to connect
0923 20:12:37.484219 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.170.190:43769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":483,"thread_start_us":367,"threads_started":1}
I20260923 20:12:37.935578 17200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:12:37.934760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.190:43769 (local address 127.1.170.131:42695)
0923 20:12:37.935330 (+ 570us) negotiation.cc:107] Waiting for socket to connect
0923 20:12:37.935411 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.170.190:43769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":458,"thread_start_us":337,"threads_started":1}
I20260923 20:12:38.017539 17201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:12:38.016758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.190:43769 (local address 127.1.170.129:44007)
0923 20:12:38.017255 (+ 497us) negotiation.cc:107] Waiting for socket to connect
0923 20:12:38.017338 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.170.190:43769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":402,"thread_start_us":299,"threads_started":1}
I20260923 20:12:38.486315 17202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:12:38.485490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.190:43769 (local address 127.1.170.130:54335)
0923 20:12:38.486038 (+ 548us) negotiation.cc:107] Waiting for socket to connect
0923 20:12:38.486134 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.170.190:43769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":430,"thread_start_us":266,"threads_started":1}
I20260923 20:12:38.938037 17203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:12:38.936646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.190:43769 (local address 127.1.170.131:58579)
0923 20:12:38.937525 (+ 879us) negotiation.cc:107] Waiting for socket to connect
0923 20:12:38.937678 (+ 153us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.170.190:43769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":711,"thread_start_us":277,"threads_started":1}
I20260923 20:12:39.019644 17204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:12:39.018788 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.190:43769 (local address 127.1.170.129:48829)
0923 20:12:39.019343 (+ 555us) negotiation.cc:107] Waiting for socket to connect
0923 20:12:39.019431 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.170.190:43769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":450,"thread_start_us":334,"threads_started":1}
I20260923 20:12:39.488214 17205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:12:39.487419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.190:43769 (local address 127.1.170.130:40155)
0923 20:12:39.487951 (+ 532us) negotiation.cc:107] Waiting for socket to connect
0923 20:12:39.488050 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.170.190:43769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":360,"thread_start_us":247,"threads_started":1}
I20260923 20:12:39.939911 17206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:12:39.939182 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.190:43769 (local address 127.1.170.131:53129)
0923 20:12:39.939635 (+ 453us) negotiation.cc:107] Waiting for socket to connect
0923 20:12:39.939729 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.170.190:43769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":330,"thread_start_us":222,"threads_started":1}
I20260923 20:12:40.022050 17207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:12:40.020796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.190:43769 (local address 127.1.170.129:32995)
0923 20:12:40.021580 (+ 784us) negotiation.cc:107] Waiting for socket to connect
0923 20:12:40.021708 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.170.190:43769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":673,"thread_start_us":359,"threads_started":1}
I20260923 20:12:40.490306 17208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:12:40.489410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.190:43769 (local address 127.1.170.130:36793)
0923 20:12:40.489966 (+ 556us) negotiation.cc:107] Waiting for socket to connect
0923 20:12:40.490106 (+ 140us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.170.190:43769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":379,"thread_start_us":238,"threads_started":1}
I20260923 20:12:40.942008 17209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:12:40.941090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.190:43769 (local address 127.1.170.131:59261)
0923 20:12:40.941620 (+ 530us) negotiation.cc:107] Waiting for socket to connect
0923 20:12:40.941724 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.170.190:43769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":410,"thread_start_us":266,"threads_started":1}
I20260923 20:12:41.024025 17210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:12:41.023016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.190:43769 (local address 127.1.170.129:34161)
0923 20:12:41.023538 (+ 522us) negotiation.cc:107] Waiting for socket to connect
0923 20:12:41.023702 (+ 164us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.170.190:43769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":407,"thread_start_us":260,"threads_started":1}
I20260923 20:12:41.492205 17211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:12:41.491450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.190:43769 (local address 127.1.170.130:40461)
0923 20:12:41.491903 (+ 453us) negotiation.cc:107] Waiting for socket to connect
0923 20:12:41.491990 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.170.190:43769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":346,"thread_start_us":240,"threads_started":1}
I20260923 20:12:41.943740 17212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:12:41.942950 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.190:43769 (local address 127.1.170.131:48635)
0923 20:12:41.943476 (+ 526us) negotiation.cc:107] Waiting for socket to connect
0923 20:12:41.943560 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.170.190:43769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":403,"thread_start_us":303,"threads_started":1}
I20260923 20:12:42.026021 17213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:12:42.025223 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.190:43769 (local address 127.1.170.129:59119)
0923 20:12:42.025671 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0923 20:12:42.025766 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.170.190:43769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":338,"thread_start_us":227,"threads_started":1}
I20260923 20:12:42.494382 17214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:12:42.493558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.190:43769 (local address 127.1.170.130:40637)
0923 20:12:42.494103 (+ 545us) negotiation.cc:107] Waiting for socket to connect
0923 20:12:42.494195 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.170.190:43769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":432,"thread_start_us":248,"threads_started":1}
I20260923 20:12:42.924026 1706 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskqUuku6/build/asan/bin/kudu pid 16742
I20260923 20:12:42.945523 17216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:12:42.944770 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.190:43769 (local address 127.1.170.131:59567)
0923 20:12:42.945248 (+ 478us) negotiation.cc:107] Waiting for socket to connect
0923 20:12:42.945335 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.170.190:43769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":348,"thread_start_us":254,"threads_started":1}
I20260923 20:12:42.948947 17215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:12:42.925179 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54032 (local address 127.1.170.129:38865)
0923 20:12:42.926245 (+ 1066us) server_negotiation.cc:206] Beginning negotiation
0923 20:12:42.926262 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0923 20:12:42.926305 (+ 43us) server_negotiation.cc:407] Connection header received
0923 20:12:42.928117 (+ 1812us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:12:42.928126 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:12:42.928520 (+ 394us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:12:42.928799 (+ 279us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:12:42.930225 (+ 1426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:12:42.931406 (+ 1181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:12:42.932656 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:12:42.933331 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:12:42.933485 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:12:42.942490 (+ 9005us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:12:42.942498 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:12:42.942516 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:12:42.942557 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:12:42.945516 (+ 2959us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:12:42.946363 (+ 847us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:12:42.946370 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:12:42.946377 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:12:42.946548 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:12:42.946947 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:12:42.946956 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:12:42.946958 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:12:42.947501 (+ 543us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:12:42.947803 (+ 302us) server_negotiation.cc:1035] Waiting for connection context
0923 20:12:42.948228 (+ 425us) server_negotiation.cc:299] Negotiation successful
0923 20:12:42.948656 (+ 428us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":907,"thread_start_us":793,"threads_started":1}
I20260923 20:12:42.950335 16900 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:12:43.036677 1706 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqUuku6/build/asan/bin/kudu with pid 16742
I20260923 20:12:43.051725 1706 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskqUuku6/build/asan/bin/kudu pid 16908
I20260923 20:12:43.070987 17219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:12:43.052868 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56188 (local address 127.1.170.130:37359)
0923 20:12:43.053474 (+ 606us) server_negotiation.cc:206] Beginning negotiation
0923 20:12:43.053488 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0923 20:12:43.053542 (+ 54us) server_negotiation.cc:407] Connection header received
0923 20:12:43.055177 (+ 1635us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:12:43.055188 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:12:43.055625 (+ 437us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:12:43.055994 (+ 369us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:12:43.057693 (+ 1699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:12:43.059240 (+ 1547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:12:43.060575 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:12:43.061271 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:12:43.061497 (+ 226us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:12:43.064411 (+ 2914us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:12:43.064419 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:12:43.064436 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:12:43.064477 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:12:43.067470 (+ 2993us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:12:43.068274 (+ 804us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:12:43.068282 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:12:43.068288 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:12:43.068456 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:12:43.068867 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:12:43.068873 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:12:43.068876 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:12:43.069427 (+ 551us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:12:43.069752 (+ 325us) server_negotiation.cc:1035] Waiting for connection context
0923 20:12:43.070262 (+ 510us) server_negotiation.cc:299] Negotiation successful
0923 20:12:43.070700 (+ 438us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":460,"thread_start_us":325,"threads_started":1}
I20260923 20:12:43.072664 17031 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:12:43.158226 1706 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqUuku6/build/asan/bin/kudu with pid 16908
I20260923 20:12:43.175868 1706 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskqUuku6/build/asan/bin/kudu pid 17039
I20260923 20:12:43.194602 17221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:12:43.176914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38614 (local address 127.1.170.131:40407)
0923 20:12:43.177492 (+ 578us) server_negotiation.cc:206] Beginning negotiation
0923 20:12:43.177506 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0923 20:12:43.177547 (+ 41us) server_negotiation.cc:407] Connection header received
0923 20:12:43.179286 (+ 1739us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:12:43.179295 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:12:43.179723 (+ 428us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:12:43.180068 (+ 345us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:12:43.181580 (+ 1512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:12:43.182867 (+ 1287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:12:43.184096 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:12:43.184730 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:12:43.184934 (+ 204us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:12:43.187594 (+ 2660us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:12:43.187604 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:12:43.187619 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:12:43.187660 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:12:43.190900 (+ 3240us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:12:43.191720 (+ 820us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:12:43.191734 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:12:43.191742 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:12:43.191920 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:12:43.192333 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:12:43.192340 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:12:43.192344 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:12:43.192929 (+ 585us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:12:43.193282 (+ 353us) server_negotiation.cc:1035] Waiting for connection context
0923 20:12:43.193798 (+ 516us) server_negotiation.cc:299] Negotiation successful
0923 20:12:43.194269 (+ 471us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":421,"thread_start_us":271,"threads_started":1}
I20260923 20:12:43.195967 17162 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:12:43.279250 1706 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqUuku6/build/asan/bin/kudu with pid 17039
2026-09-23T20:12:43Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (24519 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194121577027-1706-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 20:12:43 dist-test-slave-th4g krb5kdc[17225](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 20:12:43 dist-test-slave-th4g krb5kdc[17225](info): set up 2 sockets
Sep 23 20:12:43 dist-test-slave-th4g krb5kdc[17225](info): commencing operation
krb5kdc: starting...
W20260923 20:12:44.418819 1706 mini_kdc.cc:121] Time spent starting KDC: real 1.098s user 0.004s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194121577027-1706-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194121577027-1706-0/krb5kdc/test-user.keytab.
Sep 23 20:12:44 dist-test-slave-th4g krb5kdc[17225](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194364, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T20:12:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T20:12:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.170.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.170.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.170.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194121577027-1706-0/krb5kdc/kudu_127.1.170.190.keytab.
Entry for principal kudu/127.1.170.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194121577027-1706-0/krb5kdc/kudu_127.1.170.190.keytab.
I20260923 20:12:44.686906 1706 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqUuku6/build/asan/bin/kudu
/tmp/dist-test-taskqUuku6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194121577027-1706-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194121577027-1706-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194121577027-1706-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194121577027-1706-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.170.190:44989
--webserver_interface=127.1.170.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194121577027-1706-0/krb5kdc/kudu_127.1.170.190.keytab
--principal=kudu/127.1.170.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.170.148:42533
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.170.190:44989
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194121577027-1706-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194121577027-1706-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194121577027-1706-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:12:45.041628 17238 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:12:45.042028 17238 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:12:45.042207 17238 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:12:45.049571 17238 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260923 20:12:45.049733 17238 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260923 20:12:45.049811 17238 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260923 20:12:45.049934 17238 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 20:12:45.050040 17238 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:12:45.050158 17238 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:12:45.050249 17238 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 20:12:45.050329 17238 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 20:12:45.061218 17238 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.170.148:42533
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194121577027-1706-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194121577027-1706-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.1.170.190:44989
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194121577027-1706-0/krb5kdc/kudu_127.1.170.190.keytab
--principal=kudu/127.1.170.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.170.190:44989
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194121577027-1706-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.170.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194121577027-1706-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:12:45.062609 17238 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:12:45.063993 17238 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:12:45.077260 17244 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:12:45.077347 17238 server_base.cc:1034] running on GCE node
W20260923 20:12:45.077219 17243 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:12:45.077625 17246 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:12:45.078617 17238 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:12:45.080084 17238 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:12:45.081321 17238 hybrid_clock.cc:648] HybridClock initialized: now 1790194365081257 us; error 26 us; skew 500 ppm
I20260923 20:12:45.081755 17238 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:12:45 dist-test-slave-th4g krb5kdc[17225](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194365, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.170.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:12:45.086283 17238 init.cc:377] Logged in from keytab as kudu/127.1.170.190@KRBTEST.COM (short username kudu)
I20260923 20:12:45.088208 17238 webserver.cc:460] Webserver started at http://127.1.170.190:34517/ using document root <none> and password file <none>
I20260923 20:12:45.088984 17238 fs_manager.cc:359] Metadata directory not provided
I20260923 20:12:45.089128 17238 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:12:45.089462 17238 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:12:45.092736 17238 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194121577027-1706-0/minicluster-data/master-0/data/instance:
uuid: "5cef025810a84435a7e1983ca1e0ac5b"
format_stamp: "Formatted at 2026-09-23 20:12:45 on dist-test-slave-th4g"
server_key: "1588b5bcbf8324cd5145e5581d5e4841"
server_key_iv: "c671879784aa95788080e2a19cc270a9"
server_key_version: "encryptionkey@0"
I20260923 20:12:45.093622 17238 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194121577027-1706-0/minicluster-data/master-0/wal/instance:
uuid: "5cef025810a84435a7e1983ca1e0ac5b"
format_stamp: "Formatted at 2026-09-23 20:12:45 on dist-test-slave-th4g"
server_key: "1588b5bcbf8324cd5145e5581d5e4841"
server_key_iv: "c671879784aa95788080e2a19cc270a9"
server_key_version: "encryptionkey@0"
I20260923 20:12:45.099205 17238 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260923 20:12:45.102978 17253 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:12:45.104542 17238 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260923 20:12:45.104758 17238 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194121577027-1706-0/minicluster-data/master-0/data,/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194121577027-1706-0/minicluster-data/master-0/wal
uuid: "5cef025810a84435a7e1983ca1e0ac5b"
format_stamp: "Formatted at 2026-09-23 20:12:45 on dist-test-slave-th4g"
server_key: "1588b5bcbf8324cd5145e5581d5e4841"
server_key_iv: "c671879784aa95788080e2a19cc270a9"
server_key_version: "encryptionkey@0"
I20260923 20:12:45.104998 17238 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194121577027-1706-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194121577027-1706-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194121577027-1706-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:12:45.138055 17238 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:12:45.142987 17238 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:12:45.143294 17238 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:12:45.162906 17238 rpc_server.cc:307] RPC server started. Bound to: 127.1.170.190:44989
I20260923 20:12:45.162916 17304 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.170.190:44989 every 8 connection(s)
I20260923 20:12:45.164325 17238 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194121577027-1706-0/minicluster-data/master-0/data/info.pb
I20260923 20:12:45.168176 17305 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:12:45.172981 1706 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqUuku6/build/asan/bin/kudu as pid 17238
I20260923 20:12:45.173228 1706 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194121577027-1706-0/minicluster-data/master-0/wal/instance
I20260923 20:12:45.174069 1706 external_mini_cluster.cc:1428] Setting key 3fa29f9695a90ee77b6fcf723774626b
I20260923 20:12:45.181618 17305 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5cef025810a84435a7e1983ca1e0ac5b: Bootstrap starting.
I20260923 20:12:45.185523 17305 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5cef025810a84435a7e1983ca1e0ac5b: Neither blocks nor log segments found. Creating new log.
I20260923 20:12:45.186743 17305 log.cc:824] T 00000000000000000000000000000000 P 5cef025810a84435a7e1983ca1e0ac5b: Log is configured to *not* fsync() on all Append() calls
I20260923 20:12:45.189762 17305 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5cef025810a84435a7e1983ca1e0ac5b: No bootstrap required, opened a new log
Sep 23 20:12:45 dist-test-slave-th4g krb5kdc[17225](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194364, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.170.190@KRBTEST.COM
I20260923 20:12:45.200410 17305 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5cef025810a84435a7e1983ca1e0ac5b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5cef025810a84435a7e1983ca1e0ac5b" member_type: VOTER last_known_addr { host: "127.1.170.190" port: 44989 } }
I20260923 20:12:45.200964 17305 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5cef025810a84435a7e1983ca1e0ac5b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:12:45.201081 17305 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5cef025810a84435a7e1983ca1e0ac5b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5cef025810a84435a7e1983ca1e0ac5b, State: Initialized, Role: FOLLOWER
I20260923 20:12:45.201776 17305 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5cef025810a84435a7e1983ca1e0ac5b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5cef025810a84435a7e1983ca1e0ac5b" member_type: VOTER last_known_addr { host: "127.1.170.190" port: 44989 } }
I20260923 20:12:45.202134 17305 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5cef025810a84435a7e1983ca1e0ac5b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 20:12:45.202282 17305 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5cef025810a84435a7e1983ca1e0ac5b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 20:12:45.202461 17305 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5cef025810a84435a7e1983ca1e0ac5b [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:12:45.202858 17308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:12:45.176632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60230 (local address 127.1.170.190:44989)
0923 20:12:45.177740 (+ 1108us) server_negotiation.cc:206] Beginning negotiation
0923 20:12:45.177756 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0923 20:12:45.178154 (+ 398us) server_negotiation.cc:407] Connection header received
0923 20:12:45.180662 (+ 2508us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:12:45.180693 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:12:45.181231 (+ 538us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:12:45.181693 (+ 462us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:12:45.183335 (+ 1642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:12:45.185329 (+ 1994us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:12:45.187062 (+ 1733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:12:45.187970 (+ 908us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:12:45.188159 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:12:45.193130 (+ 4971us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:12:45.193146 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:12:45.193173 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:12:45.193238 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:12:45.197745 (+ 4507us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:12:45.198706 (+ 961us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:12:45.198716 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:12:45.198727 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:12:45.198959 (+ 232us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:12:45.199434 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:12:45.199442 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:12:45.199448 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:12:45.200201 (+ 753us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:12:45.200519 (+ 318us) server_negotiation.cc:1035] Waiting for connection context
0923 20:12:45.200902 (+ 383us) server_negotiation.cc:299] Negotiation successful
0923 20:12:45.201393 (+ 491us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":802,"thread_start_us":503,"threads_started":1}
I20260923 20:12:45.205518 17305 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5cef025810a84435a7e1983ca1e0ac5b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5cef025810a84435a7e1983ca1e0ac5b" member_type: VOTER last_known_addr { host: "127.1.170.190" port: 44989 } }
I20260923 20:12:45.206148 17305 leader_election.cc:302] T 00000000000000000000000000000000 P 5cef025810a84435a7e1983ca1e0ac5b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5cef025810a84435a7e1983ca1e0ac5b; no voters:
I20260923 20:12:45.206734 17305 leader_election.cc:288] T 00000000000000000000000000000000 P 5cef025810a84435a7e1983ca1e0ac5b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 20:12:45.206897 17310 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5cef025810a84435a7e1983ca1e0ac5b [term 1 FOLLOWER]: Leader election won for term 1
I20260923 20:12:45.207890 17310 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5cef025810a84435a7e1983ca1e0ac5b [term 1 LEADER]: Becoming Leader. State: Replica: 5cef025810a84435a7e1983ca1e0ac5b, State: Running, Role: LEADER
I20260923 20:12:45.208443 17310 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5cef025810a84435a7e1983ca1e0ac5b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5cef025810a84435a7e1983ca1e0ac5b" member_type: VOTER last_known_addr { host: "127.1.170.190" port: 44989 } }
I20260923 20:12:45.209357 17305 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5cef025810a84435a7e1983ca1e0ac5b [sys.catalog]: configured and running, proceeding with master startup.
I20260923 20:12:45.211393 17312 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5cef025810a84435a7e1983ca1e0ac5b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5cef025810a84435a7e1983ca1e0ac5b. Latest consensus state: current_term: 1 leader_uuid: "5cef025810a84435a7e1983ca1e0ac5b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5cef025810a84435a7e1983ca1e0ac5b" member_type: VOTER last_known_addr { host: "127.1.170.190" port: 44989 } } }
I20260923 20:12:45.211496 17311 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5cef025810a84435a7e1983ca1e0ac5b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5cef025810a84435a7e1983ca1e0ac5b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5cef025810a84435a7e1983ca1e0ac5b" member_type: VOTER last_known_addr { host: "127.1.170.190" port: 44989 } } }
I20260923 20:12:45.211809 17312 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5cef025810a84435a7e1983ca1e0ac5b [sys.catalog]: This master's current role is: LEADER
I20260923 20:12:45.211934 17311 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5cef025810a84435a7e1983ca1e0ac5b [sys.catalog]: This master's current role is: LEADER
I20260923 20:12:45.212951 17315 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 20:12:45.220141 17315 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 20:12:45.230902 17315 catalog_manager.cc:1347] Generated new cluster ID: e6eac8c738fb459ca0542b13a8d2eb00
I20260923 20:12:45.231062 17315 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 20:12:45.244578 17315 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 20:12:45.246588 17315 catalog_manager.cc:1504] Loading token signing keys...
I20260923 20:12:45.259271 17315 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5cef025810a84435a7e1983ca1e0ac5b: Generated new TSK 0
I20260923 20:12:45.260416 17315 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.170.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.170.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.170.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194121577027-1706-0/krb5kdc/kudu_127.1.170.129.keytab.
Entry for principal kudu/127.1.170.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194121577027-1706-0/krb5kdc/kudu_127.1.170.129.keytab.
I20260923 20:12:45.330926 1706 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqUuku6/build/asan/bin/kudu
/tmp/dist-test-taskqUuku6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194121577027-1706-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194121577027-1706-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194121577027-1706-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194121577027-1706-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.170.129:0
--local_ip_for_outbound_sockets=127.1.170.129
--webserver_interface=127.1.170.129
--webserver_port=0
--tserver_master_addrs=127.1.170.190:44989
--keytab_file=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194121577027-1706-0/krb5kdc/kudu_127.1.170.129.keytab
--principal=kudu/127.1.170.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.170.148:42533
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194121577027-1706-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194121577027-1706-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194121577027-1706-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:12:45.664855 17331 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:12:45.665148 17331 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:12:45.665306 17331 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:12:45.672912 17331 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:12:45.673024 17331 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:12:45.673188 17331 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.170.129
I20260923 20:12:45.683145 17331 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.170.148:42533
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194121577027-1706-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194121577027-1706-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194121577027-1706-0/krb5kdc/kudu_127.1.170.129.keytab
--principal=kudu/127.1.170.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.170.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194121577027-1706-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.170.129
--webserver_port=0
--tserver_master_addrs=127.1.170.190:44989
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.170.129
--log_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194121577027-1706-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:12:45.684460 17331 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:12:45.685690 17331 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:12:45.699064 17337 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:12:45.699106 17336 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:12:45.699131 17339 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:12:45.699925 17331 server_base.cc:1034] running on GCE node
I20260923 20:12:45.700675 17331 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:12:45.701742 17331 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:12:45.703075 17331 hybrid_clock.cc:648] HybridClock initialized: now 1790194365703032 us; error 29 us; skew 500 ppm
I20260923 20:12:45.703413 17331 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:12:45 dist-test-slave-th4g krb5kdc[17225](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194365, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.170.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:12:45.706836 17331 init.cc:377] Logged in from keytab as kudu/127.1.170.129@KRBTEST.COM (short username kudu)
I20260923 20:12:45.708297 17331 webserver.cc:460] Webserver started at http://127.1.170.129:36989/ using document root <none> and password file <none>
I20260923 20:12:45.709108 17331 fs_manager.cc:359] Metadata directory not provided
I20260923 20:12:45.709264 17331 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:12:45.709585 17331 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:12:45.712579 17331 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194121577027-1706-0/minicluster-data/ts-0/data/instance:
uuid: "da1071f8ba4c4d93b8a8d1d6f143babe"
format_stamp: "Formatted at 2026-09-23 20:12:45 on dist-test-slave-th4g"
server_key: "e2497ed057070bf253ec73cee9ca10f9"
server_key_iv: "781814aef2a88d65a7440c984ab50e6c"
server_key_version: "encryptionkey@0"
I20260923 20:12:45.713346 17331 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194121577027-1706-0/minicluster-data/ts-0/wal/instance:
uuid: "da1071f8ba4c4d93b8a8d1d6f143babe"
format_stamp: "Formatted at 2026-09-23 20:12:45 on dist-test-slave-th4g"
server_key: "e2497ed057070bf253ec73cee9ca10f9"
server_key_iv: "781814aef2a88d65a7440c984ab50e6c"
server_key_version: "encryptionkey@0"
I20260923 20:12:45.718873 17331 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260923 20:12:45.722353 17346 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:12:45.723682 17331 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260923 20:12:45.723869 17331 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194121577027-1706-0/minicluster-data/ts-0/data,/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194121577027-1706-0/minicluster-data/ts-0/wal
uuid: "da1071f8ba4c4d93b8a8d1d6f143babe"
format_stamp: "Formatted at 2026-09-23 20:12:45 on dist-test-slave-th4g"
server_key: "e2497ed057070bf253ec73cee9ca10f9"
server_key_iv: "781814aef2a88d65a7440c984ab50e6c"
server_key_version: "encryptionkey@0"
I20260923 20:12:45.724073 17331 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194121577027-1706-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194121577027-1706-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194121577027-1706-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:12:45.744613 17331 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:12:45.749270 17331 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:12:45.749603 17331 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:12:45.750757 17331 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 20:12:45.752194 17331 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 20:12:45.752302 17331 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 20:12:45.752451 17331 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 20:12:45.752525 17331 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 20:12:45.790933 17331 rpc_server.cc:307] RPC server started. Bound to: 127.1.170.129:43223
I20260923 20:12:45.791025 17458 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.170.129:43223 every 8 connection(s)
I20260923 20:12:45.792235 17331 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194121577027-1706-0/minicluster-data/ts-0/data/info.pb
I20260923 20:12:45.794952 1706 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqUuku6/build/asan/bin/kudu as pid 17331
I20260923 20:12:45.795186 1706 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194121577027-1706-0/minicluster-data/ts-0/wal/instance
I20260923 20:12:45.795980 1706 external_mini_cluster.cc:1428] Setting key c86354fa7d2d21d879c659e4c3e03ad3
Sep 23 20:12:45 dist-test-slave-th4g krb5kdc[17225](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194365, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.170.129@KRBTEST.COM for kudu/127.1.170.190@KRBTEST.COM
I20260923 20:12:45.818198 17461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:12:45.795207 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.170.129:55209 (local address 127.1.170.190:44989)
0923 20:12:45.795723 (+ 516us) server_negotiation.cc:206] Beginning negotiation
0923 20:12:45.795727 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 20:12:45.797539 (+ 1812us) server_negotiation.cc:407] Connection header received
0923 20:12:45.799945 (+ 2406us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:12:45.799955 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:12:45.800059 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:12:45.800190 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:12:45.801894 (+ 1704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:12:45.803205 (+ 1311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:12:45.804934 (+ 1729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:12:45.805754 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:12:45.805953 (+ 199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:12:45.811117 (+ 5164us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:12:45.811123 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:12:45.811127 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:12:45.811167 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:12:45.814144 (+ 2977us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:12:45.815151 (+ 1007us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:12:45.815159 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:12:45.815163 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:12:45.815303 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:12:45.815862 (+ 559us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:12:45.815870 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:12:45.815884 (+ 14us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:12:45.816311 (+ 427us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:12:45.816486 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0923 20:12:45.817266 (+ 780us) server_negotiation.cc:299] Negotiation successful
0923 20:12:45.817740 (+ 474us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":398,"thread_start_us":276,"threads_started":1}
I20260923 20:12:45.819090 17462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:12:45.795828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.190:44989 (local address 127.1.170.129:55209)
0923 20:12:45.797359 (+ 1531us) negotiation.cc:107] Waiting for socket to connect
0923 20:12:45.797413 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0923 20:12:45.799623 (+ 2210us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:12:45.800437 (+ 814us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:12:45.800447 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:12:45.801045 (+ 598us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:12:45.801670 (+ 625us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:12:45.801688 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:12:45.803416 (+ 1728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:12:45.803421 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:12:45.804760 (+ 1339us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:12:45.804774 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:12:45.806112 (+ 1338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:12:45.806117 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:12:45.806407 (+ 290us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:12:45.807277 (+ 870us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:12:45.807323 (+ 46us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:12:45.810802 (+ 3479us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:12:45.814388 (+ 3586us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:12:45.814397 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:12:45.814413 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:12:45.815012 (+ 599us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:12:45.815451 (+ 439us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:12:45.815456 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:12:45.815460 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:12:45.815695 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:12:45.816449 (+ 754us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:12:45.816456 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:12:45.816846 (+ 390us) client_negotiation.cc:769] Sending connection context
0923 20:12:45.817101 (+ 255us) client_negotiation.cc:240] Negotiation successful
0923 20:12:45.817655 (+ 554us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1231,"thread_start_us":1003,"threads_started":1}
I20260923 20:12:45.820920 17459 heartbeater.cc:347] Connected to a master server at 127.1.170.190:44989
I20260923 20:12:45.821280 17459 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.1.170.130@KRBTEST.COM; defaulting to no policy
I20260923 20:12:45.822185 17459 heartbeater.cc:511] Master 127.1.170.190:44989 requested a full tablet report, sending...
I20260923 20:12:45.824317 17270 ts_manager.cc:194] Registered new tserver with Master: da1071f8ba4c4d93b8a8d1d6f143babe (127.1.170.129:43223)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.170.130@KRBTEST.COM" created.
I20260923 20:12:45.825996 17270 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.170.129@KRBTEST.COM'} at 127.1.170.129:55209
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.170.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194121577027-1706-0/krb5kdc/kudu_127.1.170.130.keytab.
Entry for principal kudu/127.1.170.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194121577027-1706-0/krb5kdc/kudu_127.1.170.130.keytab.
I20260923 20:12:45.858686 1706 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqUuku6/build/asan/bin/kudu
/tmp/dist-test-taskqUuku6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194121577027-1706-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194121577027-1706-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194121577027-1706-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194121577027-1706-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.170.130:0
--local_ip_for_outbound_sockets=127.1.170.130
--webserver_interface=127.1.170.130
--webserver_port=0
--tserver_master_addrs=127.1.170.190:44989
--keytab_file=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194121577027-1706-0/krb5kdc/kudu_127.1.170.130.keytab
--principal=kudu/127.1.170.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.170.148:42533
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194121577027-1706-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194121577027-1706-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194121577027-1706-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:12:46.198947 17465 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:12:46.199225 17465 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:12:46.199379 17465 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:12:46.206696 17465 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:12:46.206802 17465 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:12:46.207007 17465 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.170.130
I20260923 20:12:46.217612 17465 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.170.148:42533
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194121577027-1706-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194121577027-1706-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194121577027-1706-0/krb5kdc/kudu_127.1.170.130.keytab
--principal=kudu/127.1.170.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.170.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194121577027-1706-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.170.130
--webserver_port=0
--tserver_master_addrs=127.1.170.190:44989
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.170.130
--log_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194121577027-1706-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:12:46.218958 17465 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:12:46.220285 17465 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:12:46.234287 17471 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:12:46.234407 17465 server_base.cc:1034] running on GCE node
W20260923 20:12:46.234287 17470 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:12:46.234699 17473 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:12:46.235380 17465 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:12:46.236379 17465 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:12:46.237649 17465 hybrid_clock.cc:648] HybridClock initialized: now 1790194366237597 us; error 48 us; skew 500 ppm
I20260923 20:12:46.238103 17465 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:12:46 dist-test-slave-th4g krb5kdc[17225](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194366, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.170.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:12:46.241310 17465 init.cc:377] Logged in from keytab as kudu/127.1.170.130@KRBTEST.COM (short username kudu)
I20260923 20:12:46.242928 17465 webserver.cc:460] Webserver started at http://127.1.170.130:39021/ using document root <none> and password file <none>
I20260923 20:12:46.243595 17465 fs_manager.cc:359] Metadata directory not provided
I20260923 20:12:46.243707 17465 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:12:46.243966 17465 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:12:46.246843 17465 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194121577027-1706-0/minicluster-data/ts-1/data/instance:
uuid: "908ce75423b74581b7328602946ef436"
format_stamp: "Formatted at 2026-09-23 20:12:46 on dist-test-slave-th4g"
server_key: "f87b9f08f4d882504e9e47a613fa64a2"
server_key_iv: "dc05933e11d906b278f9186a65b2404a"
server_key_version: "encryptionkey@0"
I20260923 20:12:46.247519 17465 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194121577027-1706-0/minicluster-data/ts-1/wal/instance:
uuid: "908ce75423b74581b7328602946ef436"
format_stamp: "Formatted at 2026-09-23 20:12:46 on dist-test-slave-th4g"
server_key: "f87b9f08f4d882504e9e47a613fa64a2"
server_key_iv: "dc05933e11d906b278f9186a65b2404a"
server_key_version: "encryptionkey@0"
I20260923 20:12:46.252584 17465 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260923 20:12:46.256031 17480 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:12:46.257437 17465 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260923 20:12:46.257575 17465 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194121577027-1706-0/minicluster-data/ts-1/data,/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194121577027-1706-0/minicluster-data/ts-1/wal
uuid: "908ce75423b74581b7328602946ef436"
format_stamp: "Formatted at 2026-09-23 20:12:46 on dist-test-slave-th4g"
server_key: "f87b9f08f4d882504e9e47a613fa64a2"
server_key_iv: "dc05933e11d906b278f9186a65b2404a"
server_key_version: "encryptionkey@0"
I20260923 20:12:46.257735 17465 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194121577027-1706-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194121577027-1706-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194121577027-1706-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:12:46.295454 17465 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:12:46.300395 17465 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:12:46.300647 17465 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:12:46.301582 17465 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 20:12:46.303022 17465 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 20:12:46.303099 17465 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 20:12:46.303188 17465 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 20:12:46.303232 17465 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 20:12:46.338965 17465 rpc_server.cc:307] RPC server started. Bound to: 127.1.170.130:35485
I20260923 20:12:46.339148 17592 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.170.130:35485 every 8 connection(s)
I20260923 20:12:46.340309 17465 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194121577027-1706-0/minicluster-data/ts-1/data/info.pb
I20260923 20:12:46.343657 1706 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqUuku6/build/asan/bin/kudu as pid 17465
I20260923 20:12:46.343883 1706 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194121577027-1706-0/minicluster-data/ts-1/wal/instance
I20260923 20:12:46.344487 1706 external_mini_cluster.cc:1428] Setting key d251b522def2a87a64b46d8c39d04e88
Sep 23 20:12:46 dist-test-slave-th4g krb5kdc[17225](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194366, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.170.130@KRBTEST.COM for kudu/127.1.170.190@KRBTEST.COM
I20260923 20:12:46.363118 17595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:12:46.342919 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.170.130:43311 (local address 127.1.170.190:44989)
0923 20:12:46.343465 (+ 546us) server_negotiation.cc:206] Beginning negotiation
0923 20:12:46.343469 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 20:12:46.344581 (+ 1112us) server_negotiation.cc:407] Connection header received
0923 20:12:46.346868 (+ 2287us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:12:46.346873 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:12:46.346959 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:12:46.347076 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:12:46.348606 (+ 1530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:12:46.349819 (+ 1213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:12:46.351527 (+ 1708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:12:46.352315 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:12:46.352504 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:12:46.356406 (+ 3902us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:12:46.356412 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:12:46.356415 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:12:46.356451 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:12:46.359585 (+ 3134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:12:46.360393 (+ 808us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:12:46.360399 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:12:46.360404 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:12:46.360542 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:12:46.360987 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:12:46.360996 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:12:46.360998 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:12:46.361311 (+ 313us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:12:46.361481 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0923 20:12:46.362320 (+ 839us) server_negotiation.cc:299] Negotiation successful
0923 20:12:46.362611 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":421,"thread_start_us":269,"threads_started":1}
I20260923 20:12:46.363832 17596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:12:46.343498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.190:44989 (local address 127.1.170.130:43311)
0923 20:12:46.344420 (+ 922us) negotiation.cc:107] Waiting for socket to connect
0923 20:12:46.344462 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0923 20:12:46.346616 (+ 2154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:12:46.347342 (+ 726us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:12:46.347353 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:12:46.347880 (+ 527us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:12:46.348425 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:12:46.348441 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:12:46.350068 (+ 1627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:12:46.350072 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:12:46.351336 (+ 1264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:12:46.351349 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:12:46.352562 (+ 1213us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:12:46.352566 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:12:46.352775 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:12:46.353558 (+ 783us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:12:46.353581 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:12:46.356140 (+ 2559us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:12:46.359768 (+ 3628us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:12:46.359779 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:12:46.359798 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:12:46.360255 (+ 457us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:12:46.360660 (+ 405us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:12:46.360665 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:12:46.360669 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:12:46.360857 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:12:46.361464 (+ 607us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:12:46.361472 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:12:46.361836 (+ 364us) client_negotiation.cc:769] Sending connection context
0923 20:12:46.362187 (+ 351us) client_negotiation.cc:240] Negotiation successful
0923 20:12:46.362700 (+ 513us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":693,"thread_start_us":507,"threads_started":1}
I20260923 20:12:46.365537 17593 heartbeater.cc:347] Connected to a master server at 127.1.170.190:44989
I20260923 20:12:46.365924 17593 heartbeater.cc:464] Registering TS with master...
I20260923 20:12:46.366784 17593 heartbeater.cc:511] Master 127.1.170.190:44989 requested a full tablet report, sending...
I20260923 20:12:46.368425 17270 ts_manager.cc:194] Registered new tserver with Master: 908ce75423b74581b7328602946ef436 (127.1.170.130:35485)
WARNING: no policy specified for kudu/127.1.170.131@KRBTEST.COM; defaulting to no policy
I20260923 20:12:46.369365 17270 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.170.130@KRBTEST.COM'} at 127.1.170.130:43311
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.170.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.170.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194121577027-1706-0/krb5kdc/kudu_127.1.170.131.keytab.
Entry for principal kudu/127.1.170.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194121577027-1706-0/krb5kdc/kudu_127.1.170.131.keytab.
I20260923 20:12:46.402128 1706 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqUuku6/build/asan/bin/kudu
/tmp/dist-test-taskqUuku6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194121577027-1706-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194121577027-1706-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194121577027-1706-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194121577027-1706-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.170.131:0
--local_ip_for_outbound_sockets=127.1.170.131
--webserver_interface=127.1.170.131
--webserver_port=0
--tserver_master_addrs=127.1.170.190:44989
--keytab_file=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194121577027-1706-0/krb5kdc/kudu_127.1.170.131.keytab
--principal=kudu/127.1.170.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.170.148:42533
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194121577027-1706-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194121577027-1706-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194121577027-1706-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:12:46.744166 17599 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:12:46.744448 17599 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:12:46.744617 17599 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:12:46.752095 17599 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:12:46.752214 17599 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:12:46.752419 17599 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.170.131
I20260923 20:12:46.762936 17599 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.170.148:42533
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194121577027-1706-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194121577027-1706-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194121577027-1706-0/krb5kdc/kudu_127.1.170.131.keytab
--principal=kudu/127.1.170.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.170.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194121577027-1706-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.170.131
--webserver_port=0
--tserver_master_addrs=127.1.170.190:44989
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.170.131
--log_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194121577027-1706-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:12:46.764199 17599 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:12:46.765491 17599 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:12:46.778448 17605 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:12:46.778514 17604 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:12:46.778420 17607 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:12:46.779362 17599 server_base.cc:1034] running on GCE node
I20260923 20:12:46.779964 17599 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:12:46.780926 17599 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:12:46.782145 17599 hybrid_clock.cc:648] HybridClock initialized: now 1790194366782083 us; error 49 us; skew 500 ppm
I20260923 20:12:46.782480 17599 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:12:46 dist-test-slave-th4g krb5kdc[17225](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194366, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.170.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:12:46.785776 17599 init.cc:377] Logged in from keytab as kudu/127.1.170.131@KRBTEST.COM (short username kudu)
I20260923 20:12:46.787274 17599 webserver.cc:460] Webserver started at http://127.1.170.131:34253/ using document root <none> and password file <none>
I20260923 20:12:46.787962 17599 fs_manager.cc:359] Metadata directory not provided
I20260923 20:12:46.788110 17599 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:12:46.788546 17599 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:12:46.791971 17599 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194121577027-1706-0/minicluster-data/ts-2/data/instance:
uuid: "b5dae6161ce245f8b7586278900d0d1e"
format_stamp: "Formatted at 2026-09-23 20:12:46 on dist-test-slave-th4g"
server_key: "4501962af6ef3f402113111651fa2957"
server_key_iv: "d7e9d4f7ce561423488988002d01dd7e"
server_key_version: "encryptionkey@0"
I20260923 20:12:46.792729 17599 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194121577027-1706-0/minicluster-data/ts-2/wal/instance:
uuid: "b5dae6161ce245f8b7586278900d0d1e"
format_stamp: "Formatted at 2026-09-23 20:12:46 on dist-test-slave-th4g"
server_key: "4501962af6ef3f402113111651fa2957"
server_key_iv: "d7e9d4f7ce561423488988002d01dd7e"
server_key_version: "encryptionkey@0"
I20260923 20:12:46.797793 17599 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260923 20:12:46.801223 17614 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:12:46.802564 17599 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260923 20:12:46.802731 17599 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194121577027-1706-0/minicluster-data/ts-2/data,/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194121577027-1706-0/minicluster-data/ts-2/wal
uuid: "b5dae6161ce245f8b7586278900d0d1e"
format_stamp: "Formatted at 2026-09-23 20:12:46 on dist-test-slave-th4g"
server_key: "4501962af6ef3f402113111651fa2957"
server_key_iv: "d7e9d4f7ce561423488988002d01dd7e"
server_key_version: "encryptionkey@0"
I20260923 20:12:46.802934 17599 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194121577027-1706-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194121577027-1706-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194121577027-1706-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:12:46.825112 17599 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:12:46.829241 17459 heartbeater.cc:503] Master 127.1.170.190:44989 was elected leader, sending a full tablet report...
I20260923 20:12:46.829962 17599 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:12:46.830251 17599 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:12:46.831476 17599 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 20:12:46.832897 17599 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 20:12:46.832978 17599 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 20:12:46.833060 17599 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 20:12:46.833104 17599 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 20:12:46.866837 17599 rpc_server.cc:307] RPC server started. Bound to: 127.1.170.131:34241
I20260923 20:12:46.866881 17726 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.170.131:34241 every 8 connection(s)
I20260923 20:12:46.868167 17599 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194121577027-1706-0/minicluster-data/ts-2/data/info.pb
I20260923 20:12:46.876715 1706 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqUuku6/build/asan/bin/kudu as pid 17599
I20260923 20:12:46.876928 1706 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194121577027-1706-0/minicluster-data/ts-2/wal/instance
I20260923 20:12:46.877437 1706 external_mini_cluster.cc:1428] Setting key 6f2bbc00dcc5156a0b393b3c7bd0037d
Sep 23 20:12:46 dist-test-slave-th4g krb5kdc[17225](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194366, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.170.131@KRBTEST.COM for kudu/127.1.170.190@KRBTEST.COM
I20260923 20:12:46.888052 17729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:12:46.870823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.170.131:35299 (local address 127.1.170.190:44989)
0923 20:12:46.871285 (+ 462us) server_negotiation.cc:206] Beginning negotiation
0923 20:12:46.871290 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 20:12:46.872133 (+ 843us) server_negotiation.cc:407] Connection header received
0923 20:12:46.874003 (+ 1870us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:12:46.874009 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:12:46.874112 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:12:46.874242 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:12:46.875707 (+ 1465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:12:46.876511 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:12:46.878406 (+ 1895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:12:46.878988 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:12:46.879088 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:12:46.882666 (+ 3578us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:12:46.882670 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:12:46.882674 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:12:46.882714 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:12:46.885211 (+ 2497us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:12:46.885983 (+ 772us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:12:46.885988 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:12:46.885992 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:12:46.886119 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:12:46.886556 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:12:46.886560 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:12:46.886562 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:12:46.886829 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:12:46.886955 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0923 20:12:46.887637 (+ 682us) server_negotiation.cc:299] Negotiation successful
0923 20:12:46.887797 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":356,"thread_start_us":257,"threads_started":1}
I20260923 20:12:46.888968 17730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:12:46.871277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.190:44989 (local address 127.1.170.131:35299)
0923 20:12:46.871963 (+ 686us) negotiation.cc:107] Waiting for socket to connect
0923 20:12:46.872004 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0923 20:12:46.873699 (+ 1695us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:12:46.874461 (+ 762us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:12:46.874469 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:12:46.874982 (+ 513us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:12:46.875525 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:12:46.875542 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:12:46.876682 (+ 1140us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:12:46.876687 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:12:46.878224 (+ 1537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:12:46.878238 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:12:46.879204 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:12:46.879209 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:12:46.879511 (+ 302us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:12:46.880238 (+ 727us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:12:46.880265 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:12:46.882474 (+ 2209us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:12:46.885337 (+ 2863us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:12:46.885343 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:12:46.885355 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:12:46.885801 (+ 446us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:12:46.886219 (+ 418us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:12:46.886222 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:12:46.886224 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:12:46.886425 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:12:46.886939 (+ 514us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:12:46.886945 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:12:46.887290 (+ 345us) client_negotiation.cc:769] Sending connection context
0923 20:12:46.887508 (+ 218us) client_negotiation.cc:240] Negotiation successful
0923 20:12:46.887888 (+ 380us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":475,"thread_start_us":275,"threads_started":1}
I20260923 20:12:46.890492 17727 heartbeater.cc:347] Connected to a master server at 127.1.170.190:44989
I20260923 20:12:46.890787 17727 heartbeater.cc:464] Registering TS with master...
I20260923 20:12:46.891556 17727 heartbeater.cc:511] Master 127.1.170.190:44989 requested a full tablet report, sending...
I20260923 20:12:46.892915 17270 ts_manager.cc:194] Registered new tserver with Master: b5dae6161ce245f8b7586278900d0d1e (127.1.170.131:34241)
I20260923 20:12:46.893702 17270 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.170.131@KRBTEST.COM'} at 127.1.170.131:35299
I20260923 20:12:46.905141 1706 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260923 20:12:46.922791 17729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:12:46.909810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60236 (local address 127.1.170.190:44989)
0923 20:12:46.910033 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0923 20:12:46.910039 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:12:46.910590 (+ 551us) server_negotiation.cc:407] Connection header received
0923 20:12:46.910781 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:12:46.910786 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:12:46.910881 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:12:46.910981 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:12:46.912443 (+ 1462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:12:46.913209 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:12:46.914705 (+ 1496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:12:46.915130 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:12:46.915233 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:12:46.917650 (+ 2417us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:12:46.917654 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:12:46.917657 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:12:46.917688 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:12:46.920389 (+ 2701us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:12:46.921067 (+ 678us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:12:46.921071 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:12:46.921074 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:12:46.921202 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:12:46.921557 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:12:46.921560 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:12:46.921562 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:12:46.921863 (+ 301us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:12:46.922110 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0923 20:12:46.922397 (+ 287us) server_negotiation.cc:299] Negotiation successful
0923 20:12:46.922577 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96}
I20260923 20:12:46.926909 17270 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60236:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 20:12:46.933840 17270 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 20:12:46.968248 17745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:12:46.959321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.129:43223 (local address 127.0.0.1:37068)
0923 20:12:46.961220 (+ 1899us) negotiation.cc:107] Waiting for socket to connect
0923 20:12:46.961241 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0923 20:12:46.961419 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:12:46.962084 (+ 665us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:12:46.962090 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:12:46.962119 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:12:46.962495 (+ 376us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:12:46.962511 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:12:46.964078 (+ 1567us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:12:46.964086 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:12:46.965639 (+ 1553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:12:46.965655 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:12:46.967273 (+ 1618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:12:46.967281 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:12:46.967474 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:12:46.967537 (+ 63us) client_negotiation.cc:769] Sending connection context
0923 20:12:46.967664 (+ 127us) client_negotiation.cc:240] Negotiation successful
0923 20:12:46.967861 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1766,"thread_start_us":1596,"threads_started":1}
I20260923 20:12:46.968843 17744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:12:46.958414 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37068 (local address 127.1.170.129:43223)
0923 20:12:46.960701 (+ 2287us) server_negotiation.cc:206] Beginning negotiation
0923 20:12:46.960711 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0923 20:12:46.961348 (+ 637us) server_negotiation.cc:407] Connection header received
0923 20:12:46.961603 (+ 255us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:12:46.961610 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:12:46.961863 (+ 253us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:12:46.962082 (+ 219us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:12:46.962651 (+ 569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:12:46.963944 (+ 1293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:12:46.965798 (+ 1854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:12:46.967057 (+ 1259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:12:46.967197 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:12:46.967433 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0923 20:12:46.968358 (+ 925us) server_negotiation.cc:299] Negotiation successful
0923 20:12:46.968525 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2117,"thread_start_us":1951,"threads_started":1}
I20260923 20:12:46.969067 17741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:12:46.946076 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.130:35485 (local address 127.0.0.1:59196)
0923 20:12:46.957438 (+ 11362us) negotiation.cc:107] Waiting for socket to connect
0923 20:12:46.957501 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0923 20:12:46.957669 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:12:46.961155 (+ 3486us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:12:46.961160 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:12:46.961192 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:12:46.961597 (+ 405us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:12:46.961609 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:12:46.963466 (+ 1857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:12:46.963471 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:12:46.966280 (+ 2809us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:12:46.966295 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:12:46.967733 (+ 1438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:12:46.967741 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:12:46.967929 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:12:46.967960 (+ 31us) client_negotiation.cc:769] Sending connection context
0923 20:12:46.968052 (+ 92us) client_negotiation.cc:240] Negotiation successful
0923 20:12:46.968193 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11082,"mutex_wait_us":167,"thread_start_us":12481,"threads_started":1}
I20260923 20:12:46.969272 17742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:12:46.946327 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59196 (local address 127.1.170.130:35485)
0923 20:12:46.960400 (+ 14073us) server_negotiation.cc:206] Beginning negotiation
0923 20:12:46.960408 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 20:12:46.960435 (+ 27us) server_negotiation.cc:407] Connection header received
0923 20:12:46.960544 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:12:46.960551 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:12:46.960854 (+ 303us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:12:46.961055 (+ 201us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:12:46.961790 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:12:46.963310 (+ 1520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:12:46.966465 (+ 3155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:12:46.967514 (+ 1049us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:12:46.967661 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:12:46.967823 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0923 20:12:46.968121 (+ 298us) server_negotiation.cc:299] Negotiation successful
0923 20:12:46.968321 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13882,"thread_start_us":13728,"threads_started":1}
I20260923 20:12:46.970678 17740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:12:46.946156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.131:34241 (local address 127.0.0.1:38882)
0923 20:12:46.958517 (+ 12361us) negotiation.cc:107] Waiting for socket to connect
0923 20:12:46.958536 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0923 20:12:46.958648 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:12:46.960318 (+ 1670us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:12:46.960328 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:12:46.960371 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:12:46.960826 (+ 455us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:12:46.960843 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:12:46.963324 (+ 2481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:12:46.963329 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:12:46.966280 (+ 2951us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:12:46.966295 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:12:46.969870 (+ 3575us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:12:46.969882 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:12:46.970080 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:12:46.970111 (+ 31us) client_negotiation.cc:769] Sending connection context
0923 20:12:46.970206 (+ 95us) client_negotiation.cc:240] Negotiation successful
0923 20:12:46.970322 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12249,"mutex_wait_us":182,"thread_start_us":10847,"threads_started":1}
I20260923 20:12:46.971388 17743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:12:46.946539 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38882 (local address 127.1.170.131:34241)
0923 20:12:46.959255 (+ 12716us) server_negotiation.cc:206] Beginning negotiation
0923 20:12:46.959268 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0923 20:12:46.959309 (+ 41us) server_negotiation.cc:407] Connection header received
0923 20:12:46.959431 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:12:46.959442 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:12:46.959765 (+ 323us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:12:46.960001 (+ 236us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:12:46.961122 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:12:46.963156 (+ 2034us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:12:46.966438 (+ 3282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:12:46.967788 (+ 1350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:12:46.967913 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:12:46.968143 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0923 20:12:46.970897 (+ 2754us) server_negotiation.cc:299] Negotiation successful
0923 20:12:46.971060 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12490,"thread_start_us":12325,"threads_started":1}
I20260923 20:12:46.972533 17528 tablet_service.cc:1467] Processing CreateTablet for tablet b462f2c7b9f94877af6d69de17d4c773 (DEFAULT_TABLE table=test-table [id=76cfc236b98b4c9ea98e86e62adbdfa2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 20:12:46.972518 17394 tablet_service.cc:1467] Processing CreateTablet for tablet b462f2c7b9f94877af6d69de17d4c773 (DEFAULT_TABLE table=test-table [id=76cfc236b98b4c9ea98e86e62adbdfa2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 20:12:46.973868 17394 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b462f2c7b9f94877af6d69de17d4c773. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:12:46.973943 17528 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b462f2c7b9f94877af6d69de17d4c773. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:12:46.974994 17662 tablet_service.cc:1467] Processing CreateTablet for tablet b462f2c7b9f94877af6d69de17d4c773 (DEFAULT_TABLE table=test-table [id=76cfc236b98b4c9ea98e86e62adbdfa2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 20:12:46.976375 17662 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b462f2c7b9f94877af6d69de17d4c773. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:12:46.985571 17746 tablet_bootstrap.cc:492] T b462f2c7b9f94877af6d69de17d4c773 P da1071f8ba4c4d93b8a8d1d6f143babe: Bootstrap starting.
I20260923 20:12:46.988794 17746 tablet_bootstrap.cc:654] T b462f2c7b9f94877af6d69de17d4c773 P da1071f8ba4c4d93b8a8d1d6f143babe: Neither blocks nor log segments found. Creating new log.
I20260923 20:12:46.989624 17747 tablet_bootstrap.cc:492] T b462f2c7b9f94877af6d69de17d4c773 P 908ce75423b74581b7328602946ef436: Bootstrap starting.
I20260923 20:12:46.990146 17746 log.cc:824] T b462f2c7b9f94877af6d69de17d4c773 P da1071f8ba4c4d93b8a8d1d6f143babe: Log is configured to *not* fsync() on all Append() calls
I20260923 20:12:46.993844 17747 tablet_bootstrap.cc:654] T b462f2c7b9f94877af6d69de17d4c773 P 908ce75423b74581b7328602946ef436: Neither blocks nor log segments found. Creating new log.
I20260923 20:12:46.994668 17748 tablet_bootstrap.cc:492] T b462f2c7b9f94877af6d69de17d4c773 P b5dae6161ce245f8b7586278900d0d1e: Bootstrap starting.
I20260923 20:12:46.995746 17747 log.cc:824] T b462f2c7b9f94877af6d69de17d4c773 P 908ce75423b74581b7328602946ef436: Log is configured to *not* fsync() on all Append() calls
I20260923 20:12:46.997534 17746 tablet_bootstrap.cc:492] T b462f2c7b9f94877af6d69de17d4c773 P da1071f8ba4c4d93b8a8d1d6f143babe: No bootstrap required, opened a new log
I20260923 20:12:46.997808 17746 ts_tablet_manager.cc:1397] T b462f2c7b9f94877af6d69de17d4c773 P da1071f8ba4c4d93b8a8d1d6f143babe: Time spent bootstrapping tablet: real 0.012s user 0.006s sys 0.003s
I20260923 20:12:46.998373 17747 tablet_bootstrap.cc:492] T b462f2c7b9f94877af6d69de17d4c773 P 908ce75423b74581b7328602946ef436: No bootstrap required, opened a new log
I20260923 20:12:46.998628 17747 ts_tablet_manager.cc:1397] T b462f2c7b9f94877af6d69de17d4c773 P 908ce75423b74581b7328602946ef436: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.001s
I20260923 20:12:46.999142 17748 tablet_bootstrap.cc:654] T b462f2c7b9f94877af6d69de17d4c773 P b5dae6161ce245f8b7586278900d0d1e: Neither blocks nor log segments found. Creating new log.
I20260923 20:12:47.000360 17748 log.cc:824] T b462f2c7b9f94877af6d69de17d4c773 P b5dae6161ce245f8b7586278900d0d1e: Log is configured to *not* fsync() on all Append() calls
I20260923 20:12:47.003719 17748 tablet_bootstrap.cc:492] T b462f2c7b9f94877af6d69de17d4c773 P b5dae6161ce245f8b7586278900d0d1e: No bootstrap required, opened a new log
I20260923 20:12:47.003980 17748 ts_tablet_manager.cc:1397] T b462f2c7b9f94877af6d69de17d4c773 P b5dae6161ce245f8b7586278900d0d1e: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.004s
I20260923 20:12:47.006597 17746 raft_consensus.cc:348] T b462f2c7b9f94877af6d69de17d4c773 P da1071f8ba4c4d93b8a8d1d6f143babe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "908ce75423b74581b7328602946ef436" member_type: VOTER last_known_addr { host: "127.1.170.130" port: 35485 } } peers { permanent_uuid: "b5dae6161ce245f8b7586278900d0d1e" member_type: VOTER last_known_addr { host: "127.1.170.131" port: 34241 } } peers { permanent_uuid: "da1071f8ba4c4d93b8a8d1d6f143babe" member_type: VOTER last_known_addr { host: "127.1.170.129" port: 43223 } }
I20260923 20:12:47.007050 17746 raft_consensus.cc:374] T b462f2c7b9f94877af6d69de17d4c773 P da1071f8ba4c4d93b8a8d1d6f143babe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:12:47.007151 17746 raft_consensus.cc:729] T b462f2c7b9f94877af6d69de17d4c773 P da1071f8ba4c4d93b8a8d1d6f143babe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: da1071f8ba4c4d93b8a8d1d6f143babe, State: Initialized, Role: FOLLOWER
I20260923 20:12:47.007771 17746 consensus_queue.cc:260] T b462f2c7b9f94877af6d69de17d4c773 P da1071f8ba4c4d93b8a8d1d6f143babe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "908ce75423b74581b7328602946ef436" member_type: VOTER last_known_addr { host: "127.1.170.130" port: 35485 } } peers { permanent_uuid: "b5dae6161ce245f8b7586278900d0d1e" member_type: VOTER last_known_addr { host: "127.1.170.131" port: 34241 } } peers { permanent_uuid: "da1071f8ba4c4d93b8a8d1d6f143babe" member_type: VOTER last_known_addr { host: "127.1.170.129" port: 43223 } }
I20260923 20:12:47.008791 17747 raft_consensus.cc:348] T b462f2c7b9f94877af6d69de17d4c773 P 908ce75423b74581b7328602946ef436 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "908ce75423b74581b7328602946ef436" member_type: VOTER last_known_addr { host: "127.1.170.130" port: 35485 } } peers { permanent_uuid: "b5dae6161ce245f8b7586278900d0d1e" member_type: VOTER last_known_addr { host: "127.1.170.131" port: 34241 } } peers { permanent_uuid: "da1071f8ba4c4d93b8a8d1d6f143babe" member_type: VOTER last_known_addr { host: "127.1.170.129" port: 43223 } }
I20260923 20:12:47.009209 17747 raft_consensus.cc:374] T b462f2c7b9f94877af6d69de17d4c773 P 908ce75423b74581b7328602946ef436 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:12:47.009300 17747 raft_consensus.cc:729] T b462f2c7b9f94877af6d69de17d4c773 P 908ce75423b74581b7328602946ef436 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 908ce75423b74581b7328602946ef436, State: Initialized, Role: FOLLOWER
I20260923 20:12:47.010128 17747 consensus_queue.cc:260] T b462f2c7b9f94877af6d69de17d4c773 P 908ce75423b74581b7328602946ef436 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "908ce75423b74581b7328602946ef436" member_type: VOTER last_known_addr { host: "127.1.170.130" port: 35485 } } peers { permanent_uuid: "b5dae6161ce245f8b7586278900d0d1e" member_type: VOTER last_known_addr { host: "127.1.170.131" port: 34241 } } peers { permanent_uuid: "da1071f8ba4c4d93b8a8d1d6f143babe" member_type: VOTER last_known_addr { host: "127.1.170.129" port: 43223 } }
I20260923 20:12:47.011479 17593 heartbeater.cc:503] Master 127.1.170.190:44989 was elected leader, sending a full tablet report...
I20260923 20:12:47.011960 17747 ts_tablet_manager.cc:1428] T b462f2c7b9f94877af6d69de17d4c773 P 908ce75423b74581b7328602946ef436: Time spent starting tablet: real 0.013s user 0.005s sys 0.007s
I20260923 20:12:47.013468 17746 ts_tablet_manager.cc:1428] T b462f2c7b9f94877af6d69de17d4c773 P da1071f8ba4c4d93b8a8d1d6f143babe: Time spent starting tablet: real 0.015s user 0.009s sys 0.006s
I20260923 20:12:47.014757 17748 raft_consensus.cc:348] T b462f2c7b9f94877af6d69de17d4c773 P b5dae6161ce245f8b7586278900d0d1e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "908ce75423b74581b7328602946ef436" member_type: VOTER last_known_addr { host: "127.1.170.130" port: 35485 } } peers { permanent_uuid: "b5dae6161ce245f8b7586278900d0d1e" member_type: VOTER last_known_addr { host: "127.1.170.131" port: 34241 } } peers { permanent_uuid: "da1071f8ba4c4d93b8a8d1d6f143babe" member_type: VOTER last_known_addr { host: "127.1.170.129" port: 43223 } }
I20260923 20:12:47.015196 17748 raft_consensus.cc:374] T b462f2c7b9f94877af6d69de17d4c773 P b5dae6161ce245f8b7586278900d0d1e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:12:47.015296 17748 raft_consensus.cc:729] T b462f2c7b9f94877af6d69de17d4c773 P b5dae6161ce245f8b7586278900d0d1e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b5dae6161ce245f8b7586278900d0d1e, State: Initialized, Role: FOLLOWER
I20260923 20:12:47.015995 17748 consensus_queue.cc:260] T b462f2c7b9f94877af6d69de17d4c773 P b5dae6161ce245f8b7586278900d0d1e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "908ce75423b74581b7328602946ef436" member_type: VOTER last_known_addr { host: "127.1.170.130" port: 35485 } } peers { permanent_uuid: "b5dae6161ce245f8b7586278900d0d1e" member_type: VOTER last_known_addr { host: "127.1.170.131" port: 34241 } } peers { permanent_uuid: "da1071f8ba4c4d93b8a8d1d6f143babe" member_type: VOTER last_known_addr { host: "127.1.170.129" port: 43223 } }
I20260923 20:12:47.017063 17727 heartbeater.cc:503] Master 127.1.170.190:44989 was elected leader, sending a full tablet report...
I20260923 20:12:47.017715 17748 ts_tablet_manager.cc:1428] T b462f2c7b9f94877af6d69de17d4c773 P b5dae6161ce245f8b7586278900d0d1e: Time spent starting tablet: real 0.014s user 0.006s sys 0.008s
W20260923 20:12:47.044770 17460 tablet.cc:2369] T b462f2c7b9f94877af6d69de17d4c773 P da1071f8ba4c4d93b8a8d1d6f143babe: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260923 20:12:47.092406 17594 tablet.cc:2369] T b462f2c7b9f94877af6d69de17d4c773 P 908ce75423b74581b7328602946ef436: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 20:12:47.100430 17752 raft_consensus.cc:482] T b462f2c7b9f94877af6d69de17d4c773 P da1071f8ba4c4d93b8a8d1d6f143babe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 20:12:47.100687 17752 raft_consensus.cc:504] T b462f2c7b9f94877af6d69de17d4c773 P da1071f8ba4c4d93b8a8d1d6f143babe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "908ce75423b74581b7328602946ef436" member_type: VOTER last_known_addr { host: "127.1.170.130" port: 35485 } } peers { permanent_uuid: "b5dae6161ce245f8b7586278900d0d1e" member_type: VOTER last_known_addr { host: "127.1.170.131" port: 34241 } } peers { permanent_uuid: "da1071f8ba4c4d93b8a8d1d6f143babe" member_type: VOTER last_known_addr { host: "127.1.170.129" port: 43223 } }
I20260923 20:12:47.102569 17752 leader_election.cc:288] T b462f2c7b9f94877af6d69de17d4c773 P da1071f8ba4c4d93b8a8d1d6f143babe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 908ce75423b74581b7328602946ef436 (127.1.170.130:35485), b5dae6161ce245f8b7586278900d0d1e (127.1.170.131:34241)
I20260923 20:12:47.112330 17743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:12:47.103238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.170.129:60763 (local address 127.1.170.131:34241)
0923 20:12:47.103753 (+ 515us) server_negotiation.cc:206] Beginning negotiation
0923 20:12:47.103762 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 20:12:47.103988 (+ 226us) server_negotiation.cc:407] Connection header received
0923 20:12:47.104090 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:12:47.104096 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:12:47.104181 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:12:47.104281 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:12:47.105444 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:12:47.106620 (+ 1176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:12:47.109484 (+ 2864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:12:47.110869 (+ 1385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:12:47.111030 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:12:47.111103 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0923 20:12:47.111919 (+ 816us) server_negotiation.cc:299] Negotiation successful
0923 20:12:47.112096 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":379}
I20260923 20:12:47.112327 17742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:12:47.102951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.170.129:40329 (local address 127.1.170.130:35485)
0923 20:12:47.103584 (+ 633us) server_negotiation.cc:206] Beginning negotiation
0923 20:12:47.103595 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0923 20:12:47.103719 (+ 124us) server_negotiation.cc:407] Connection header received
0923 20:12:47.104091 (+ 372us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:12:47.104097 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:12:47.104178 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:12:47.104281 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:12:47.105278 (+ 997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:12:47.106613 (+ 1335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:12:47.109491 (+ 2878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:12:47.110790 (+ 1299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:12:47.110990 (+ 200us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:12:47.111072 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0923 20:12:47.111919 (+ 847us) server_negotiation.cc:299] Negotiation successful
0923 20:12:47.112054 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":383}
I20260923 20:12:47.112633 17755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:12:47.102761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.130:35485 (local address 127.1.170.129:40329)
0923 20:12:47.103594 (+ 833us) negotiation.cc:107] Waiting for socket to connect
0923 20:12:47.103619 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0923 20:12:47.103849 (+ 230us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:12:47.104354 (+ 505us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:12:47.104360 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:12:47.104408 (+ 48us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:12:47.105086 (+ 678us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:12:47.105107 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:12:47.106779 (+ 1672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:12:47.106785 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:12:47.109299 (+ 2514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:12:47.109314 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:12:47.111315 (+ 2001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:12:47.111323 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:12:47.111642 (+ 319us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:12:47.111748 (+ 106us) client_negotiation.cc:769] Sending connection context
0923 20:12:47.111864 (+ 116us) client_negotiation.cc:240] Negotiation successful
0923 20:12:47.112049 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":667,"mutex_wait_us":218,"spinlock_wait_cycles":9344,"thread_start_us":454,"threads_started":1}
I20260923 20:12:47.112656 17756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:12:47.102883 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.131:34241 (local address 127.1.170.129:60763)
0923 20:12:47.103849 (+ 966us) negotiation.cc:107] Waiting for socket to connect
0923 20:12:47.103866 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0923 20:12:47.103980 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:12:47.104354 (+ 374us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:12:47.104360 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:12:47.104401 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:12:47.105245 (+ 844us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:12:47.105257 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:12:47.106779 (+ 1522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:12:47.106785 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:12:47.109309 (+ 2524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:12:47.109318 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:12:47.111329 (+ 2011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:12:47.111342 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:12:47.111637 (+ 295us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:12:47.111749 (+ 112us) client_negotiation.cc:769] Sending connection context
0923 20:12:47.111864 (+ 115us) client_negotiation.cc:240] Negotiation successful
0923 20:12:47.112049 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":857,"mutex_wait_us":362,"spinlock_wait_cycles":42752,"thread_start_us":684,"threads_started":1}
I20260923 20:12:47.113698 17548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b462f2c7b9f94877af6d69de17d4c773" candidate_uuid: "da1071f8ba4c4d93b8a8d1d6f143babe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "908ce75423b74581b7328602946ef436" is_pre_election: true
I20260923 20:12:47.113865 17682 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b462f2c7b9f94877af6d69de17d4c773" candidate_uuid: "da1071f8ba4c4d93b8a8d1d6f143babe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b5dae6161ce245f8b7586278900d0d1e" is_pre_election: true
I20260923 20:12:47.114362 17548 raft_consensus.cc:2454] T b462f2c7b9f94877af6d69de17d4c773 P 908ce75423b74581b7328602946ef436 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate da1071f8ba4c4d93b8a8d1d6f143babe in term 0.
I20260923 20:12:47.114442 17682 raft_consensus.cc:2454] T b462f2c7b9f94877af6d69de17d4c773 P b5dae6161ce245f8b7586278900d0d1e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate da1071f8ba4c4d93b8a8d1d6f143babe in term 0.
I20260923 20:12:47.115127 17350 leader_election.cc:302] T b462f2c7b9f94877af6d69de17d4c773 P da1071f8ba4c4d93b8a8d1d6f143babe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 908ce75423b74581b7328602946ef436, b5dae6161ce245f8b7586278900d0d1e, da1071f8ba4c4d93b8a8d1d6f143babe; no voters:
I20260923 20:12:47.115576 17752 raft_consensus.cc:2781] T b462f2c7b9f94877af6d69de17d4c773 P da1071f8ba4c4d93b8a8d1d6f143babe [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 20:12:47.115742 17752 raft_consensus.cc:482] T b462f2c7b9f94877af6d69de17d4c773 P da1071f8ba4c4d93b8a8d1d6f143babe [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 20:12:47.115906 17752 raft_consensus.cc:3037] T b462f2c7b9f94877af6d69de17d4c773 P da1071f8ba4c4d93b8a8d1d6f143babe [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:12:47.118402 17752 raft_consensus.cc:504] T b462f2c7b9f94877af6d69de17d4c773 P da1071f8ba4c4d93b8a8d1d6f143babe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "908ce75423b74581b7328602946ef436" member_type: VOTER last_known_addr { host: "127.1.170.130" port: 35485 } } peers { permanent_uuid: "b5dae6161ce245f8b7586278900d0d1e" member_type: VOTER last_known_addr { host: "127.1.170.131" port: 34241 } } peers { permanent_uuid: "da1071f8ba4c4d93b8a8d1d6f143babe" member_type: VOTER last_known_addr { host: "127.1.170.129" port: 43223 } }
I20260923 20:12:47.119153 17752 leader_election.cc:288] T b462f2c7b9f94877af6d69de17d4c773 P da1071f8ba4c4d93b8a8d1d6f143babe [CANDIDATE]: Term 1 election: Requested vote from peers 908ce75423b74581b7328602946ef436 (127.1.170.130:35485), b5dae6161ce245f8b7586278900d0d1e (127.1.170.131:34241)
I20260923 20:12:47.119531 17548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b462f2c7b9f94877af6d69de17d4c773" candidate_uuid: "da1071f8ba4c4d93b8a8d1d6f143babe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "908ce75423b74581b7328602946ef436"
I20260923 20:12:47.119750 17548 raft_consensus.cc:3037] T b462f2c7b9f94877af6d69de17d4c773 P 908ce75423b74581b7328602946ef436 [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:12:47.119724 17682 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b462f2c7b9f94877af6d69de17d4c773" candidate_uuid: "da1071f8ba4c4d93b8a8d1d6f143babe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b5dae6161ce245f8b7586278900d0d1e"
W20260923 20:12:47.119951 17728 tablet.cc:2369] T b462f2c7b9f94877af6d69de17d4c773 P b5dae6161ce245f8b7586278900d0d1e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 20:12:47.119958 17682 raft_consensus.cc:3037] T b462f2c7b9f94877af6d69de17d4c773 P b5dae6161ce245f8b7586278900d0d1e [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:12:47.122454 17548 raft_consensus.cc:2454] T b462f2c7b9f94877af6d69de17d4c773 P 908ce75423b74581b7328602946ef436 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate da1071f8ba4c4d93b8a8d1d6f143babe in term 1.
I20260923 20:12:47.122550 17682 raft_consensus.cc:2454] T b462f2c7b9f94877af6d69de17d4c773 P b5dae6161ce245f8b7586278900d0d1e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate da1071f8ba4c4d93b8a8d1d6f143babe in term 1.
I20260923 20:12:47.122929 17350 leader_election.cc:302] T b462f2c7b9f94877af6d69de17d4c773 P da1071f8ba4c4d93b8a8d1d6f143babe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 908ce75423b74581b7328602946ef436, da1071f8ba4c4d93b8a8d1d6f143babe; no voters:
I20260923 20:12:47.123288 17752 raft_consensus.cc:2781] T b462f2c7b9f94877af6d69de17d4c773 P da1071f8ba4c4d93b8a8d1d6f143babe [term 1 FOLLOWER]: Leader election won for term 1
I20260923 20:12:47.123767 17752 raft_consensus.cc:686] T b462f2c7b9f94877af6d69de17d4c773 P da1071f8ba4c4d93b8a8d1d6f143babe [term 1 LEADER]: Becoming Leader. State: Replica: da1071f8ba4c4d93b8a8d1d6f143babe, State: Running, Role: LEADER
I20260923 20:12:47.124276 17752 consensus_queue.cc:237] T b462f2c7b9f94877af6d69de17d4c773 P da1071f8ba4c4d93b8a8d1d6f143babe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "908ce75423b74581b7328602946ef436" member_type: VOTER last_known_addr { host: "127.1.170.130" port: 35485 } } peers { permanent_uuid: "b5dae6161ce245f8b7586278900d0d1e" member_type: VOTER last_known_addr { host: "127.1.170.131" port: 34241 } } peers { permanent_uuid: "da1071f8ba4c4d93b8a8d1d6f143babe" member_type: VOTER last_known_addr { host: "127.1.170.129" port: 43223 } }
I20260923 20:12:47.129014 17270 catalog_manager.cc:5495] T b462f2c7b9f94877af6d69de17d4c773 P da1071f8ba4c4d93b8a8d1d6f143babe reported cstate change: term changed from 0 to 1, leader changed from <none> to da1071f8ba4c4d93b8a8d1d6f143babe (127.1.170.129). New cstate: current_term: 1 leader_uuid: "da1071f8ba4c4d93b8a8d1d6f143babe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "908ce75423b74581b7328602946ef436" member_type: VOTER last_known_addr { host: "127.1.170.130" port: 35485 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b5dae6161ce245f8b7586278900d0d1e" member_type: VOTER last_known_addr { host: "127.1.170.131" port: 34241 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "da1071f8ba4c4d93b8a8d1d6f143babe" member_type: VOTER last_known_addr { host: "127.1.170.129" port: 43223 } health_report { overall_health: HEALTHY } } }
I20260923 20:12:47.149253 17744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:12:47.143817 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37076 (local address 127.1.170.129:43223)
0923 20:12:47.144017 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0923 20:12:47.144023 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:12:47.144042 (+ 19us) server_negotiation.cc:407] Connection header received
0923 20:12:47.144239 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:12:47.144243 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:12:47.144324 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:12:47.144491 (+ 167us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 20:12:47.144870 (+ 379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:12:47.145593 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:12:47.147101 (+ 1508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:12:47.147515 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:12:47.147651 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:12:47.148147 (+ 496us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 20:12:47.148666 (+ 519us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 20:12:47.148825 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0923 20:12:47.148903 (+ 78us) server_negotiation.cc:299] Negotiation successful
0923 20:12:47.148984 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260923 20:12:47.157796 17548 raft_consensus.cc:1261] T b462f2c7b9f94877af6d69de17d4c773 P 908ce75423b74581b7328602946ef436 [term 1 FOLLOWER]: Refusing update from remote peer da1071f8ba4c4d93b8a8d1d6f143babe: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 20:12:47.157796 17682 raft_consensus.cc:1261] T b462f2c7b9f94877af6d69de17d4c773 P b5dae6161ce245f8b7586278900d0d1e [term 1 FOLLOWER]: Refusing update from remote peer da1071f8ba4c4d93b8a8d1d6f143babe: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 20:12:47.158883 17757 consensus_queue.cc:1035] T b462f2c7b9f94877af6d69de17d4c773 P da1071f8ba4c4d93b8a8d1d6f143babe [LEADER]: Connected to new peer: Peer: permanent_uuid: "908ce75423b74581b7328602946ef436" member_type: VOTER last_known_addr { host: "127.1.170.130" port: 35485 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 20:12:47.159315 17752 consensus_queue.cc:1035] T b462f2c7b9f94877af6d69de17d4c773 P da1071f8ba4c4d93b8a8d1d6f143babe [LEADER]: Connected to new peer: Peer: permanent_uuid: "b5dae6161ce245f8b7586278900d0d1e" member_type: VOTER last_known_addr { host: "127.1.170.131" port: 34241 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 20:12:47.177736 17762 mvcc.cc:205] Tried to move back new op lower bound from 7332636127867006976 to 7332636127744647168. Current Snapshot: MvccSnapshot[applied={T|T < 7332636127867006976}]
I20260923 20:12:47.198045 17270 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60236:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 20:12:47.198333 17270 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60236:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 20:12:47.202479 17270 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5cef025810a84435a7e1983ca1e0ac5b: Sending DeleteTablet for 3 replicas of tablet b462f2c7b9f94877af6d69de17d4c773
I20260923 20:12:47.203735 17528 tablet_service.cc:1514] Processing DeleteTablet for tablet b462f2c7b9f94877af6d69de17d4c773 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 20:12:47 UTC) from {username='kudu', principal='kudu/127.1.170.190@KRBTEST.COM'} at 127.0.0.1:59196
I20260923 20:12:47.203926 1706 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskqUuku6/build/asan/bin/kudu pid 17331
I20260923 20:12:47.203853 17662 tablet_service.cc:1514] Processing DeleteTablet for tablet b462f2c7b9f94877af6d69de17d4c773 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 20:12:47 UTC) from {username='kudu', principal='kudu/127.1.170.190@KRBTEST.COM'} at 127.0.0.1:38882
I20260923 20:12:47.203917 17394 tablet_service.cc:1514] Processing DeleteTablet for tablet b462f2c7b9f94877af6d69de17d4c773 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 20:12:47 UTC) from {username='kudu', principal='kudu/127.1.170.190@KRBTEST.COM'} at 127.0.0.1:37068
I20260923 20:12:47.205659 17771 tablet_replica.cc:330] T b462f2c7b9f94877af6d69de17d4c773 P b5dae6161ce245f8b7586278900d0d1e: stopping tablet replica
I20260923 20:12:47.206676 17770 tablet_replica.cc:330] T b462f2c7b9f94877af6d69de17d4c773 P da1071f8ba4c4d93b8a8d1d6f143babe: stopping tablet replica
I20260923 20:12:47.207182 17770 raft_consensus.cc:2229] T b462f2c7b9f94877af6d69de17d4c773 P da1071f8ba4c4d93b8a8d1d6f143babe [term 1 LEADER]: Raft consensus shutting down.
I20260923 20:12:47.207974 17771 raft_consensus.cc:2229] T b462f2c7b9f94877af6d69de17d4c773 P b5dae6161ce245f8b7586278900d0d1e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 20:12:47.208077 17770 raft_consensus.cc:2258] T b462f2c7b9f94877af6d69de17d4c773 P da1071f8ba4c4d93b8a8d1d6f143babe [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 20:12:47.208868 17771 raft_consensus.cc:2258] T b462f2c7b9f94877af6d69de17d4c773 P b5dae6161ce245f8b7586278900d0d1e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 20:12:47.208964 17769 tablet_replica.cc:330] T b462f2c7b9f94877af6d69de17d4c773 P 908ce75423b74581b7328602946ef436: stopping tablet replica
I20260923 20:12:47.209554 17769 raft_consensus.cc:2229] T b462f2c7b9f94877af6d69de17d4c773 P 908ce75423b74581b7328602946ef436 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 20:12:47.210336 17769 raft_consensus.cc:2258] T b462f2c7b9f94877af6d69de17d4c773 P 908ce75423b74581b7328602946ef436 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 20:12:47.210573 17770 ts_tablet_manager.cc:1905] T b462f2c7b9f94877af6d69de17d4c773 P da1071f8ba4c4d93b8a8d1d6f143babe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 20:12:47.212033 17771 ts_tablet_manager.cc:1905] T b462f2c7b9f94877af6d69de17d4c773 P b5dae6161ce245f8b7586278900d0d1e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 20:12:47.212484 17769 ts_tablet_manager.cc:1905] T b462f2c7b9f94877af6d69de17d4c773 P 908ce75423b74581b7328602946ef436: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 20:12:47.218113 17770 ts_tablet_manager.cc:1918] T b462f2c7b9f94877af6d69de17d4c773 P da1071f8ba4c4d93b8a8d1d6f143babe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 20:12:47.218297 17770 log.cc:1196] T b462f2c7b9f94877af6d69de17d4c773 P da1071f8ba4c4d93b8a8d1d6f143babe: Deleting WAL directory at /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194121577027-1706-0/minicluster-data/ts-0/wal/wals/b462f2c7b9f94877af6d69de17d4c773
I20260923 20:12:47.218837 17770 ts_tablet_manager.cc:1939] T b462f2c7b9f94877af6d69de17d4c773 P da1071f8ba4c4d93b8a8d1d6f143babe: Deleting consensus metadata
I20260923 20:12:47.220165 17769 ts_tablet_manager.cc:1918] T b462f2c7b9f94877af6d69de17d4c773 P 908ce75423b74581b7328602946ef436: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 20:12:47.220399 17769 log.cc:1196] T b462f2c7b9f94877af6d69de17d4c773 P 908ce75423b74581b7328602946ef436: Deleting WAL directory at /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194121577027-1706-0/minicluster-data/ts-1/wal/wals/b462f2c7b9f94877af6d69de17d4c773
I20260923 20:12:47.220773 17254 catalog_manager.cc:4841] TS da1071f8ba4c4d93b8a8d1d6f143babe (127.1.170.129:43223): tablet b462f2c7b9f94877af6d69de17d4c773 (table test-table [id=76cfc236b98b4c9ea98e86e62adbdfa2]) successfully deleted
I20260923 20:12:47.221184 17769 ts_tablet_manager.cc:1939] T b462f2c7b9f94877af6d69de17d4c773 P 908ce75423b74581b7328602946ef436: Deleting consensus metadata
I20260923 20:12:47.223093 17257 catalog_manager.cc:4841] TS 908ce75423b74581b7328602946ef436 (127.1.170.130:35485): tablet b462f2c7b9f94877af6d69de17d4c773 (table test-table [id=76cfc236b98b4c9ea98e86e62adbdfa2]) successfully deleted
Sep 23 20:12:47 dist-test-slave-th4g krb5kdc[17225](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194364, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.170.129@KRBTEST.COM
I20260923 20:12:47.226936 17771 ts_tablet_manager.cc:1918] T b462f2c7b9f94877af6d69de17d4c773 P b5dae6161ce245f8b7586278900d0d1e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 20:12:47.227181 17771 log.cc:1196] T b462f2c7b9f94877af6d69de17d4c773 P b5dae6161ce245f8b7586278900d0d1e: Deleting WAL directory at /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194121577027-1706-0/minicluster-data/ts-2/wal/wals/b462f2c7b9f94877af6d69de17d4c773
I20260923 20:12:47.227715 17771 ts_tablet_manager.cc:1939] T b462f2c7b9f94877af6d69de17d4c773 P b5dae6161ce245f8b7586278900d0d1e: Deleting consensus metadata
I20260923 20:12:47.229759 17254 catalog_manager.cc:4841] TS b5dae6161ce245f8b7586278900d0d1e (127.1.170.131:34241): tablet b462f2c7b9f94877af6d69de17d4c773 (table test-table [id=76cfc236b98b4c9ea98e86e62adbdfa2]) successfully deleted
I20260923 20:12:47.234858 17744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:12:47.204845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37080 (local address 127.1.170.129:43223)
0923 20:12:47.205071 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0923 20:12:47.205081 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0923 20:12:47.206353 (+ 1272us) server_negotiation.cc:407] Connection header received
0923 20:12:47.206958 (+ 605us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:12:47.206964 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:12:47.207059 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:12:47.207142 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:12:47.213216 (+ 6074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:12:47.214586 (+ 1370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:12:47.219063 (+ 4477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:12:47.219949 (+ 886us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:12:47.220072 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:12:47.228041 (+ 7969us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:12:47.228056 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:12:47.228080 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:12:47.228155 (+ 75us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:12:47.231867 (+ 3712us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:12:47.232739 (+ 872us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:12:47.232747 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:12:47.232749 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:12:47.232917 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:12:47.233305 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:12:47.233308 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:12:47.233310 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:12:47.233675 (+ 365us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:12:47.234054 (+ 379us) server_negotiation.cc:1035] Waiting for connection context
0923 20:12:47.234320 (+ 266us) server_negotiation.cc:299] Negotiation successful
0923 20:12:47.234520 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260923 20:12:47.235606 17454 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:12:47.336869 1706 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqUuku6/build/asan/bin/kudu with pid 17331
I20260923 20:12:47.355064 1706 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskqUuku6/build/asan/bin/kudu pid 17465
Sep 23 20:12:47 dist-test-slave-th4g krb5kdc[17225](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194364, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.170.130@KRBTEST.COM
I20260923 20:12:47.370718 17742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:12:47.355930 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59208 (local address 127.1.170.130:35485)
0923 20:12:47.356107 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0923 20:12:47.356111 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 20:12:47.356127 (+ 16us) server_negotiation.cc:407] Connection header received
0923 20:12:47.356180 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:12:47.356183 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:12:47.356240 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:12:47.356326 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:12:47.357668 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:12:47.358469 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:12:47.359890 (+ 1421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:12:47.360273 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:12:47.360366 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:12:47.364881 (+ 4515us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:12:47.364895 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:12:47.364910 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:12:47.364942 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:12:47.367859 (+ 2917us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:12:47.368618 (+ 759us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:12:47.368625 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:12:47.368627 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:12:47.368751 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:12:47.369145 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:12:47.369149 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:12:47.369151 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:12:47.369553 (+ 402us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:12:47.369772 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0923 20:12:47.370274 (+ 502us) server_negotiation.cc:299] Negotiation successful
0923 20:12:47.370450 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260923 20:12:47.371450 17588 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:12:47.463702 1706 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqUuku6/build/asan/bin/kudu with pid 17465
I20260923 20:12:47.480527 1706 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskqUuku6/build/asan/bin/kudu pid 17599
Sep 23 20:12:47 dist-test-slave-th4g krb5kdc[17225](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194364, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.170.131@KRBTEST.COM
I20260923 20:12:47.495594 17743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:12:47.481337 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38884 (local address 127.1.170.131:34241)
0923 20:12:47.481515 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0923 20:12:47.481519 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 20:12:47.481539 (+ 20us) server_negotiation.cc:407] Connection header received
0923 20:12:47.481707 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:12:47.481711 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:12:47.481772 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:12:47.481860 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:12:47.483149 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:12:47.483866 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:12:47.485157 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:12:47.485527 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:12:47.485629 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:12:47.490050 (+ 4421us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:12:47.490076 (+ 26us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:12:47.490092 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:12:47.490127 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:12:47.492966 (+ 2839us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:12:47.493615 (+ 649us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:12:47.493622 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:12:47.493624 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:12:47.493756 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:12:47.494268 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:12:47.494272 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:12:47.494274 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:12:47.494660 (+ 386us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:12:47.494880 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0923 20:12:47.495164 (+ 284us) server_negotiation.cc:299] Negotiation successful
0923 20:12:47.495329 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260923 20:12:47.496308 17722 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:12:47.588142 1706 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqUuku6/build/asan/bin/kudu with pid 17599
I20260923 20:12:47.605536 1706 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskqUuku6/build/asan/bin/kudu pid 17238
I20260923 20:12:47.606838 17300 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:12:47.719555 1706 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqUuku6/build/asan/bin/kudu with pid 17238
I20260923 20:13:07.742275 1706 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqUuku6/build/asan/bin/kudu
/tmp/dist-test-taskqUuku6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194121577027-1706-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194121577027-1706-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194121577027-1706-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194121577027-1706-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.170.190:44989
--webserver_interface=127.1.170.190
--webserver_port=34517
--keytab_file=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194121577027-1706-0/krb5kdc/kudu_127.1.170.190.keytab
--principal=kudu/127.1.170.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.170.148:42533
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.170.190:44989
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194121577027-1706-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194121577027-1706-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194121577027-1706-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:13:08.088258 17777 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:13:08.088585 17777 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:13:08.088760 17777 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:13:08.096382 17777 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260923 20:13:08.096511 17777 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260923 20:13:08.096591 17777 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260923 20:13:08.096670 17777 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 20:13:08.096740 17777 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:13:08.096839 17777 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:13:08.096926 17777 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 20:13:08.097008 17777 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 20:13:08.107594 17777 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.170.148:42533
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194121577027-1706-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194121577027-1706-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.1.170.190:44989
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194121577027-1706-0/krb5kdc/kudu_127.1.170.190.keytab
--principal=kudu/127.1.170.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.170.190:44989
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194121577027-1706-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.170.190
--webserver_port=34517
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194121577027-1706-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:13:08.109155 17777 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:13:08.110486 17777 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:13:08.136580 17784 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:13:08.136857 17783 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:13:08.136685 17786 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:13:08.137017 17777 server_base.cc:1034] running on GCE node
I20260923 20:13:08.138286 17777 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:13:08.140015 17777 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:13:08.141309 17777 hybrid_clock.cc:648] HybridClock initialized: now 1790194388141223 us; error 52 us; skew 500 ppm
I20260923 20:13:08.141762 17777 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:13:08 dist-test-slave-th4g krb5kdc[17225](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194388, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.170.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:13:08.146070 17777 init.cc:377] Logged in from keytab as kudu/127.1.170.190@KRBTEST.COM (short username kudu)
I20260923 20:13:08.147792 17777 webserver.cc:460] Webserver started at http://127.1.170.190:34517/ using document root <none> and password file <none>
I20260923 20:13:08.148620 17777 fs_manager.cc:359] Metadata directory not provided
I20260923 20:13:08.148775 17777 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:13:08.155974 17777 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.002s sys 0.003s
I20260923 20:13:08.159451 17793 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:13:08.161315 17777 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260923 20:13:08.161557 17777 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194121577027-1706-0/minicluster-data/master-0/data,/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194121577027-1706-0/minicluster-data/master-0/wal
uuid: "5cef025810a84435a7e1983ca1e0ac5b"
format_stamp: "Formatted at 2026-09-23 20:12:45 on dist-test-slave-th4g"
server_key: "1588b5bcbf8324cd5145e5581d5e4841"
server_key_iv: "c671879784aa95788080e2a19cc270a9"
server_key_version: "encryptionkey@0"
I20260923 20:13:08.162789 17777 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194121577027-1706-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194121577027-1706-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194121577027-1706-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:13:08.181993 17777 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:13:08.187534 17777 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:13:08.187899 17777 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:13:08.211464 17777 rpc_server.cc:307] RPC server started. Bound to: 127.1.170.190:44989
I20260923 20:13:08.211503 17844 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.170.190:44989 every 8 connection(s)
I20260923 20:13:08.213100 17777 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194121577027-1706-0/minicluster-data/master-0/data/info.pb
I20260923 20:13:08.220497 1706 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqUuku6/build/asan/bin/kudu as pid 17777
I20260923 20:13:08.220909 17845 sys_catalog.cc:263] Verifying existing consensus state
I20260923 20:13:08.221562 1706 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqUuku6/build/asan/bin/kudu
/tmp/dist-test-taskqUuku6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194121577027-1706-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194121577027-1706-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194121577027-1706-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194121577027-1706-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.170.129:43223
--local_ip_for_outbound_sockets=127.1.170.129
--tserver_master_addrs=127.1.170.190:44989
--webserver_port=36989
--webserver_interface=127.1.170.129
--keytab_file=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194121577027-1706-0/krb5kdc/kudu_127.1.170.129.keytab
--principal=kudu/127.1.170.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.170.148:42533
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194121577027-1706-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194121577027-1706-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194121577027-1706-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260923 20:13:08.226507 17845 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5cef025810a84435a7e1983ca1e0ac5b: Bootstrap starting.
I20260923 20:13:08.248706 17845 log.cc:824] T 00000000000000000000000000000000 P 5cef025810a84435a7e1983ca1e0ac5b: Log is configured to *not* fsync() on all Append() calls
I20260923 20:13:08.261716 17845 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5cef025810a84435a7e1983ca1e0ac5b: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260923 20:13:08.262380 17845 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5cef025810a84435a7e1983ca1e0ac5b: Bootstrap complete.
I20260923 20:13:08.272454 17845 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5cef025810a84435a7e1983ca1e0ac5b [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5cef025810a84435a7e1983ca1e0ac5b" member_type: VOTER last_known_addr { host: "127.1.170.190" port: 44989 } }
I20260923 20:13:08.273362 17845 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5cef025810a84435a7e1983ca1e0ac5b [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5cef025810a84435a7e1983ca1e0ac5b, State: Initialized, Role: FOLLOWER
I20260923 20:13:08.274200 17845 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5cef025810a84435a7e1983ca1e0ac5b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5cef025810a84435a7e1983ca1e0ac5b" member_type: VOTER last_known_addr { host: "127.1.170.190" port: 44989 } }
I20260923 20:13:08.274628 17845 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5cef025810a84435a7e1983ca1e0ac5b [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 20:13:08.274801 17845 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5cef025810a84435a7e1983ca1e0ac5b [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 20:13:08.275008 17845 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5cef025810a84435a7e1983ca1e0ac5b [term 1 FOLLOWER]: Advancing to term 2
I20260923 20:13:08.279145 17845 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5cef025810a84435a7e1983ca1e0ac5b [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5cef025810a84435a7e1983ca1e0ac5b" member_type: VOTER last_known_addr { host: "127.1.170.190" port: 44989 } }
I20260923 20:13:08.279722 17845 leader_election.cc:302] T 00000000000000000000000000000000 P 5cef025810a84435a7e1983ca1e0ac5b [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5cef025810a84435a7e1983ca1e0ac5b; no voters:
I20260923 20:13:08.280423 17845 leader_election.cc:288] T 00000000000000000000000000000000 P 5cef025810a84435a7e1983ca1e0ac5b [CANDIDATE]: Term 2 election: Requested vote from peers
I20260923 20:13:08.280572 17849 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5cef025810a84435a7e1983ca1e0ac5b [term 2 FOLLOWER]: Leader election won for term 2
I20260923 20:13:08.283167 17845 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5cef025810a84435a7e1983ca1e0ac5b [sys.catalog]: configured and running, proceeding with master startup.
I20260923 20:13:08.287771 17849 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5cef025810a84435a7e1983ca1e0ac5b [term 2 LEADER]: Becoming Leader. State: Replica: 5cef025810a84435a7e1983ca1e0ac5b, State: Running, Role: LEADER
I20260923 20:13:08.288554 17849 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5cef025810a84435a7e1983ca1e0ac5b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5cef025810a84435a7e1983ca1e0ac5b" member_type: VOTER last_known_addr { host: "127.1.170.190" port: 44989 } }
I20260923 20:13:08.291613 17851 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5cef025810a84435a7e1983ca1e0ac5b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5cef025810a84435a7e1983ca1e0ac5b. Latest consensus state: current_term: 2 leader_uuid: "5cef025810a84435a7e1983ca1e0ac5b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5cef025810a84435a7e1983ca1e0ac5b" member_type: VOTER last_known_addr { host: "127.1.170.190" port: 44989 } } }
I20260923 20:13:08.292121 17851 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5cef025810a84435a7e1983ca1e0ac5b [sys.catalog]: This master's current role is: LEADER
I20260923 20:13:08.292632 17850 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5cef025810a84435a7e1983ca1e0ac5b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "5cef025810a84435a7e1983ca1e0ac5b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5cef025810a84435a7e1983ca1e0ac5b" member_type: VOTER last_known_addr { host: "127.1.170.190" port: 44989 } } }
I20260923 20:13:08.293010 17850 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5cef025810a84435a7e1983ca1e0ac5b [sys.catalog]: This master's current role is: LEADER
I20260923 20:13:08.301301 17860 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 20:13:08.310063 17860 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 20:13:08.311585 17860 catalog_manager.cc:1259] Loaded cluster ID: e6eac8c738fb459ca0542b13a8d2eb00
I20260923 20:13:08.311702 17860 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 20:13:08.316557 17860 catalog_manager.cc:1504] Loading token signing keys...
I20260923 20:13:08.318328 17860 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 5cef025810a84435a7e1983ca1e0ac5b: Loaded TSK: 0
I20260923 20:13:08.351395 17860 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5cef025810a84435a7e1983ca1e0ac5b: Generated new TSK 1
I20260923 20:13:08.351914 17860 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260923 20:13:08.578593 17847 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:13:08.578891 17847 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:13:08.579056 17847 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:13:08.588053 17847 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:13:08.588195 17847 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:13:08.588388 17847 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.170.129
I20260923 20:13:08.599732 17847 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.170.148:42533
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194121577027-1706-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194121577027-1706-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194121577027-1706-0/krb5kdc/kudu_127.1.170.129.keytab
--principal=kudu/127.1.170.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.170.129:43223
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194121577027-1706-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.170.129
--webserver_port=36989
--tserver_master_addrs=127.1.170.190:44989
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.170.129
--log_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194121577027-1706-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:13:08.601221 17847 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:13:08.602602 17847 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:13:08.616791 17873 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:13:08.616974 17875 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:13:08.617305 17872 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:13:08.617748 17847 server_base.cc:1034] running on GCE node
I20260923 20:13:08.618443 17847 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:13:08.619563 17847 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:13:08.620819 17847 hybrid_clock.cc:648] HybridClock initialized: now 1790194388620738 us; error 69 us; skew 500 ppm
I20260923 20:13:08.621222 17847 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:13:08 dist-test-slave-th4g krb5kdc[17225](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194388, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.170.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:13:08.625830 17847 init.cc:377] Logged in from keytab as kudu/127.1.170.129@KRBTEST.COM (short username kudu)
I20260923 20:13:08.627800 17847 webserver.cc:460] Webserver started at http://127.1.170.129:36989/ using document root <none> and password file <none>
I20260923 20:13:08.628549 17847 fs_manager.cc:359] Metadata directory not provided
I20260923 20:13:08.628719 17847 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:13:08.634923 17847 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260923 20:13:08.638162 17882 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:13:08.640395 17847 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260923 20:13:08.640714 17847 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194121577027-1706-0/minicluster-data/ts-0/data,/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194121577027-1706-0/minicluster-data/ts-0/wal
uuid: "da1071f8ba4c4d93b8a8d1d6f143babe"
format_stamp: "Formatted at 2026-09-23 20:12:45 on dist-test-slave-th4g"
server_key: "e2497ed057070bf253ec73cee9ca10f9"
server_key_iv: "781814aef2a88d65a7440c984ab50e6c"
server_key_version: "encryptionkey@0"
I20260923 20:13:08.642150 17847 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194121577027-1706-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194121577027-1706-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194121577027-1706-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:13:08.674530 17847 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:13:08.679376 17847 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:13:08.679675 17847 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:13:08.680706 17847 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 20:13:08.682277 17847 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 20:13:08.682369 17847 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 20:13:08.682531 17847 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 20:13:08.682634 17847 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 20:13:08.724309 17847 rpc_server.cc:307] RPC server started. Bound to: 127.1.170.129:43223
I20260923 20:13:08.724309 17994 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.170.129:43223 every 8 connection(s)
I20260923 20:13:08.726437 17847 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194121577027-1706-0/minicluster-data/ts-0/data/info.pb
I20260923 20:13:08.730589 1706 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqUuku6/build/asan/bin/kudu as pid 17847
I20260923 20:13:08.731359 1706 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqUuku6/build/asan/bin/kudu
/tmp/dist-test-taskqUuku6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194121577027-1706-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194121577027-1706-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194121577027-1706-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194121577027-1706-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.170.130:35485
--local_ip_for_outbound_sockets=127.1.170.130
--tserver_master_addrs=127.1.170.190:44989
--webserver_port=39021
--webserver_interface=127.1.170.130
--keytab_file=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194121577027-1706-0/krb5kdc/kudu_127.1.170.130.keytab
--principal=kudu/127.1.170.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.170.148:42533
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194121577027-1706-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194121577027-1706-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194121577027-1706-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 23 20:13:08 dist-test-slave-th4g krb5kdc[17225](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194388, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.170.129@KRBTEST.COM for kudu/127.1.170.190@KRBTEST.COM
I20260923 20:13:08.757520 17997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:13:08.729991 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.170.129:53631 (local address 127.1.170.190:44989)
0923 20:13:08.730738 (+ 747us) server_negotiation.cc:206] Beginning negotiation
0923 20:13:08.730756 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0923 20:13:08.731755 (+ 999us) server_negotiation.cc:407] Connection header received
0923 20:13:08.735027 (+ 3272us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:13:08.735040 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:13:08.735416 (+ 376us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:13:08.735722 (+ 306us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:13:08.738129 (+ 2407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:13:08.739975 (+ 1846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:13:08.741724 (+ 1749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:13:08.742959 (+ 1235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:13:08.743213 (+ 254us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:13:08.747739 (+ 4526us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:13:08.747748 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:13:08.747764 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:13:08.747813 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:13:08.751674 (+ 3861us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:13:08.753036 (+ 1362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:13:08.753045 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:13:08.753051 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:13:08.753273 (+ 222us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:13:08.753976 (+ 703us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:13:08.753980 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:13:08.753982 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:13:08.754496 (+ 514us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:13:08.754661 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0923 20:13:08.755949 (+ 1288us) server_negotiation.cc:299] Negotiation successful
0923 20:13:08.756498 (+ 549us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":515,"thread_start_us":319,"threads_started":1}
I20260923 20:13:08.757848 17998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:13:08.730482 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.190:44989 (local address 127.1.170.129:53631)
0923 20:13:08.731393 (+ 911us) negotiation.cc:107] Waiting for socket to connect
0923 20:13:08.731455 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0923 20:13:08.734356 (+ 2901us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:13:08.735977 (+ 1621us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:13:08.735991 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:13:08.736823 (+ 832us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:13:08.737606 (+ 783us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:13:08.737634 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:13:08.740222 (+ 2588us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:13:08.740227 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:13:08.741523 (+ 1296us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:13:08.741548 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:13:08.743239 (+ 1691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:13:08.743244 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:13:08.743480 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:13:08.744254 (+ 774us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:13:08.744279 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:13:08.747354 (+ 3075us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:13:08.751940 (+ 4586us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:13:08.751955 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:13:08.751978 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:13:08.752784 (+ 806us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:13:08.753435 (+ 651us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:13:08.753440 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:13:08.753447 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:13:08.753737 (+ 290us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:13:08.754663 (+ 926us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:13:08.754675 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:13:08.755204 (+ 529us) client_negotiation.cc:769] Sending connection context
0923 20:13:08.755581 (+ 377us) client_negotiation.cc:240] Negotiation successful
0923 20:13:08.756309 (+ 728us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":583,"thread_start_us":371,"threads_started":1}
I20260923 20:13:08.760994 17995 heartbeater.cc:347] Connected to a master server at 127.1.170.190:44989
I20260923 20:13:08.761608 17995 heartbeater.cc:464] Registering TS with master...
I20260923 20:13:08.762833 17995 heartbeater.cc:511] Master 127.1.170.190:44989 requested a full tablet report, sending...
I20260923 20:13:08.765398 17810 ts_manager.cc:194] Registered new tserver with Master: da1071f8ba4c4d93b8a8d1d6f143babe (127.1.170.129:43223)
I20260923 20:13:08.767732 17810 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.170.129@KRBTEST.COM'} at 127.1.170.129:53631
W20260923 20:13:09.100900 17999 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:13:09.101214 17999 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:13:09.101366 17999 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:13:09.109637 17999 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:13:09.109768 17999 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:13:09.109995 17999 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.170.130
I20260923 20:13:09.120260 17999 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.170.148:42533
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194121577027-1706-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194121577027-1706-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194121577027-1706-0/krb5kdc/kudu_127.1.170.130.keytab
--principal=kudu/127.1.170.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.170.130:35485
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194121577027-1706-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.170.130
--webserver_port=39021
--tserver_master_addrs=127.1.170.190:44989
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.170.130
--log_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194121577027-1706-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:13:09.121635 17999 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:13:09.123080 17999 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:13:09.137723 18004 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:13:09.138019 18005 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:13:09.137686 18007 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:13:09.138307 17999 server_base.cc:1034] running on GCE node
I20260923 20:13:09.139019 17999 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:13:09.140182 17999 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:13:09.141469 17999 hybrid_clock.cc:648] HybridClock initialized: now 1790194389141415 us; error 62 us; skew 500 ppm
I20260923 20:13:09.141846 17999 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:13:09 dist-test-slave-th4g krb5kdc[17225](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194389, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.170.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:13:09.145309 17999 init.cc:377] Logged in from keytab as kudu/127.1.170.130@KRBTEST.COM (short username kudu)
I20260923 20:13:09.147111 17999 webserver.cc:460] Webserver started at http://127.1.170.130:39021/ using document root <none> and password file <none>
I20260923 20:13:09.147832 17999 fs_manager.cc:359] Metadata directory not provided
I20260923 20:13:09.148003 17999 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:13:09.154771 17999 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.001s
I20260923 20:13:09.158303 18014 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:13:09.160053 17999 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260923 20:13:09.160313 17999 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194121577027-1706-0/minicluster-data/ts-1/data,/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194121577027-1706-0/minicluster-data/ts-1/wal
uuid: "908ce75423b74581b7328602946ef436"
format_stamp: "Formatted at 2026-09-23 20:12:46 on dist-test-slave-th4g"
server_key: "f87b9f08f4d882504e9e47a613fa64a2"
server_key_iv: "dc05933e11d906b278f9186a65b2404a"
server_key_version: "encryptionkey@0"
I20260923 20:13:09.161456 17999 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194121577027-1706-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194121577027-1706-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194121577027-1706-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:13:09.206313 17999 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:13:09.212436 17999 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:13:09.212749 17999 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:13:09.213820 17999 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 20:13:09.215243 17999 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 20:13:09.215379 17999 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 20:13:09.215517 17999 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 20:13:09.215603 17999 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 20:13:09.256469 17999 rpc_server.cc:307] RPC server started. Bound to: 127.1.170.130:35485
I20260923 20:13:09.256524 18126 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.170.130:35485 every 8 connection(s)
I20260923 20:13:09.258031 17999 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194121577027-1706-0/minicluster-data/ts-1/data/info.pb
I20260923 20:13:09.262876 1706 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqUuku6/build/asan/bin/kudu as pid 17999
I20260923 20:13:09.263542 1706 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqUuku6/build/asan/bin/kudu
/tmp/dist-test-taskqUuku6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194121577027-1706-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194121577027-1706-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194121577027-1706-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194121577027-1706-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.170.131:34241
--local_ip_for_outbound_sockets=127.1.170.131
--tserver_master_addrs=127.1.170.190:44989
--webserver_port=34253
--webserver_interface=127.1.170.131
--keytab_file=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194121577027-1706-0/krb5kdc/kudu_127.1.170.131.keytab
--principal=kudu/127.1.170.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.170.148:42533
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194121577027-1706-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194121577027-1706-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194121577027-1706-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 23 20:13:09 dist-test-slave-th4g krb5kdc[17225](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194389, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.170.130@KRBTEST.COM for kudu/127.1.170.190@KRBTEST.COM
I20260923 20:13:09.280480 18129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:13:09.260828 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.170.130:60411 (local address 127.1.170.190:44989)
0923 20:13:09.261337 (+ 509us) server_negotiation.cc:206] Beginning negotiation
0923 20:13:09.261341 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 20:13:09.262237 (+ 896us) server_negotiation.cc:407] Connection header received
0923 20:13:09.264237 (+ 2000us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:13:09.264244 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:13:09.264361 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:13:09.264486 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:13:09.266111 (+ 1625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:13:09.266970 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:13:09.268666 (+ 1696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:13:09.269088 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:13:09.269230 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:13:09.273850 (+ 4620us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:13:09.273856 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:13:09.273859 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:13:09.273987 (+ 128us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:13:09.276698 (+ 2711us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:13:09.277778 (+ 1080us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:13:09.277782 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:13:09.277785 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:13:09.278011 (+ 226us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:13:09.278568 (+ 557us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:13:09.278572 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:13:09.278574 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:13:09.278909 (+ 335us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:13:09.279090 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0923 20:13:09.280024 (+ 934us) server_negotiation.cc:299] Negotiation successful
0923 20:13:09.280198 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":402,"thread_start_us":312,"threads_started":1}
I20260923 20:13:09.282063 18130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:13:09.261246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.190:44989 (local address 127.1.170.130:60411)
0923 20:13:09.262073 (+ 827us) negotiation.cc:107] Waiting for socket to connect
0923 20:13:09.262120 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0923 20:13:09.263966 (+ 1846us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:13:09.264730 (+ 764us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:13:09.264738 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:13:09.265290 (+ 552us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:13:09.265932 (+ 642us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:13:09.265963 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:13:09.267125 (+ 1162us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:13:09.267131 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:13:09.268505 (+ 1374us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:13:09.268518 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:13:09.269317 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:13:09.269325 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:13:09.269596 (+ 271us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:13:09.270387 (+ 791us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:13:09.270423 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:13:09.273534 (+ 3111us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:13:09.276866 (+ 3332us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:13:09.276879 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:13:09.276901 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:13:09.277607 (+ 706us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:13:09.278136 (+ 529us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:13:09.278144 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:13:09.278150 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:13:09.278411 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:13:09.279081 (+ 670us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:13:09.279092 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:13:09.279574 (+ 482us) client_negotiation.cc:769] Sending connection context
0923 20:13:09.279881 (+ 307us) client_negotiation.cc:240] Negotiation successful
0923 20:13:09.280502 (+ 621us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":502,"thread_start_us":302,"threads_started":1}
I20260923 20:13:09.284036 18127 heartbeater.cc:347] Connected to a master server at 127.1.170.190:44989
I20260923 20:13:09.284576 18127 heartbeater.cc:464] Registering TS with master...
I20260923 20:13:09.285775 18127 heartbeater.cc:511] Master 127.1.170.190:44989 requested a full tablet report, sending...
I20260923 20:13:09.287709 17810 ts_manager.cc:194] Registered new tserver with Master: 908ce75423b74581b7328602946ef436 (127.1.170.130:35485)
I20260923 20:13:09.288568 17810 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.170.130@KRBTEST.COM'} at 127.1.170.130:60411
W20260923 20:13:09.609939 18131 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:13:09.610280 18131 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:13:09.610446 18131 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:13:09.618134 18131 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:13:09.618296 18131 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:13:09.618472 18131 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.170.131
I20260923 20:13:09.628542 18131 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.170.148:42533
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194121577027-1706-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194121577027-1706-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194121577027-1706-0/krb5kdc/kudu_127.1.170.131.keytab
--principal=kudu/127.1.170.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.170.131:34241
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194121577027-1706-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.170.131
--webserver_port=34253
--tserver_master_addrs=127.1.170.190:44989
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.170.131
--log_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194121577027-1706-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:13:09.630002 18131 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:13:09.631327 18131 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:13:09.647888 18139 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:13:09.648196 18137 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:13:09.648228 18136 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:13:09.649415 18131 server_base.cc:1034] running on GCE node
I20260923 20:13:09.650530 18131 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:13:09.651865 18131 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:13:09.653155 18131 hybrid_clock.cc:648] HybridClock initialized: now 1790194389653088 us; error 62 us; skew 500 ppm
I20260923 20:13:09.653540 18131 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:13:09 dist-test-slave-th4g krb5kdc[17225](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194389, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.170.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:13:09.657270 18131 init.cc:377] Logged in from keytab as kudu/127.1.170.131@KRBTEST.COM (short username kudu)
I20260923 20:13:09.659032 18131 webserver.cc:460] Webserver started at http://127.1.170.131:34253/ using document root <none> and password file <none>
I20260923 20:13:09.659814 18131 fs_manager.cc:359] Metadata directory not provided
I20260923 20:13:09.659988 18131 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:13:09.666182 18131 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260923 20:13:09.669992 18146 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:13:09.672210 18131 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260923 20:13:09.672488 18131 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194121577027-1706-0/minicluster-data/ts-2/data,/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194121577027-1706-0/minicluster-data/ts-2/wal
uuid: "b5dae6161ce245f8b7586278900d0d1e"
format_stamp: "Formatted at 2026-09-23 20:12:46 on dist-test-slave-th4g"
server_key: "4501962af6ef3f402113111651fa2957"
server_key_iv: "d7e9d4f7ce561423488988002d01dd7e"
server_key_version: "encryptionkey@0"
I20260923 20:13:09.673791 18131 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194121577027-1706-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194121577027-1706-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194121577027-1706-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:13:09.696547 18131 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:13:09.701272 18131 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:13:09.701594 18131 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:13:09.702725 18131 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 20:13:09.704293 18131 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 20:13:09.704394 18131 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 20:13:09.704481 18131 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 20:13:09.704535 18131 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 20:13:09.745764 18131 rpc_server.cc:307] RPC server started. Bound to: 127.1.170.131:34241
I20260923 20:13:09.745806 18258 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.170.131:34241 every 8 connection(s)
I20260923 20:13:09.747695 18131 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194121577027-1706-0/minicluster-data/ts-2/data/info.pb
I20260923 20:13:09.752632 1706 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqUuku6/build/asan/bin/kudu as pid 18131
Sep 23 20:13:09 dist-test-slave-th4g krb5kdc[17225](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194389, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.170.131@KRBTEST.COM for kudu/127.1.170.190@KRBTEST.COM
I20260923 20:13:09.771211 17995 heartbeater.cc:503] Master 127.1.170.190:44989 was elected leader, sending a full tablet report...
I20260923 20:13:09.774809 18262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:13:09.754142 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52680 (local address 127.1.170.190:44989)
0923 20:13:09.754671 (+ 529us) server_negotiation.cc:206] Beginning negotiation
0923 20:13:09.754677 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:13:09.755251 (+ 574us) server_negotiation.cc:407] Connection header received
0923 20:13:09.755442 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:13:09.755448 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:13:09.755567 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:13:09.755696 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:13:09.758600 (+ 2904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:13:09.759914 (+ 1314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:13:09.762207 (+ 2293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:13:09.762944 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:13:09.763079 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:13:09.767217 (+ 4138us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:13:09.767223 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:13:09.767226 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:13:09.767263 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:13:09.771135 (+ 3872us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:13:09.772019 (+ 884us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:13:09.772025 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:13:09.772029 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:13:09.772175 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:13:09.772592 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:13:09.772597 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:13:09.772602 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:13:09.773018 (+ 416us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:13:09.773308 (+ 290us) server_negotiation.cc:1035] Waiting for connection context
0923 20:13:09.773652 (+ 344us) server_negotiation.cc:299] Negotiation successful
0923 20:13:09.774330 (+ 678us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":67,"server-negotiator.queue_time_us":404,"thread_start_us":271,"threads_started":1}
I20260923 20:13:09.778968 18129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:13:09.750957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.170.131:37725 (local address 127.1.170.190:44989)
0923 20:13:09.751214 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0923 20:13:09.751218 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 20:13:09.752548 (+ 1330us) server_negotiation.cc:407] Connection header received
0923 20:13:09.755744 (+ 3196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:13:09.755749 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:13:09.755918 (+ 169us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:13:09.756032 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:13:09.758521 (+ 2489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:13:09.759968 (+ 1447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:13:09.762457 (+ 2489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:13:09.763097 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:13:09.763176 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:13:09.769278 (+ 6102us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:13:09.769295 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:13:09.769305 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:13:09.769363 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:13:09.774805 (+ 5442us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:13:09.775920 (+ 1115us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:13:09.775927 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:13:09.775930 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:13:09.776264 (+ 334us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:13:09.776839 (+ 575us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:13:09.776843 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:13:09.776845 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:13:09.777243 (+ 398us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:13:09.777437 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0923 20:13:09.778337 (+ 900us) server_negotiation.cc:299] Negotiation successful
0923 20:13:09.778591 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":7,"server-negotiator.queue_time_us":89}
I20260923 20:13:09.780071 18261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:13:09.751444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.190:44989 (local address 127.1.170.131:37725)
0923 20:13:09.752349 (+ 905us) negotiation.cc:107] Waiting for socket to connect
0923 20:13:09.752412 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0923 20:13:09.755333 (+ 2921us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:13:09.756372 (+ 1039us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:13:09.756400 (+ 28us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:13:09.757284 (+ 884us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:13:09.758262 (+ 978us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:13:09.758296 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:13:09.760197 (+ 1901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:13:09.760204 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:13:09.762307 (+ 2103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:13:09.762323 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:13:09.763764 (+ 1441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:13:09.763787 (+ 23us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:13:09.764506 (+ 719us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:13:09.765443 (+ 937us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:13:09.765484 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:13:09.768576 (+ 3092us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:13:09.774986 (+ 6410us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:13:09.775001 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:13:09.775023 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:13:09.775677 (+ 654us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:13:09.776424 (+ 747us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:13:09.776429 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:13:09.776432 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:13:09.776696 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:13:09.777385 (+ 689us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:13:09.777395 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:13:09.777886 (+ 491us) client_negotiation.cc:769] Sending connection context
0923 20:13:09.778241 (+ 355us) client_negotiation.cc:240] Negotiation successful
0923 20:13:09.778853 (+ 612us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":500,"thread_start_us":299,"threads_started":1}
I20260923 20:13:09.781493 18259 heartbeater.cc:347] Connected to a master server at 127.1.170.190:44989
I20260923 20:13:09.781817 18259 heartbeater.cc:464] Registering TS with master...
I20260923 20:13:09.782640 18259 heartbeater.cc:511] Master 127.1.170.190:44989 requested a full tablet report, sending...
I20260923 20:13:09.784194 17810 ts_manager.cc:194] Registered new tserver with Master: b5dae6161ce245f8b7586278900d0d1e (127.1.170.131:34241)
I20260923 20:13:09.785085 17810 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.170.131@KRBTEST.COM'} at 127.1.170.131:37725
I20260923 20:13:09.791254 1706 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260923 20:13:09.800251 18129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:13:09.792339 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52694 (local address 127.1.170.190:44989)
0923 20:13:09.792551 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0923 20:13:09.792556 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 20:13:09.793139 (+ 583us) server_negotiation.cc:407] Connection header received
0923 20:13:09.793468 (+ 329us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:13:09.793476 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:13:09.793827 (+ 351us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:13:09.794152 (+ 325us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 20:13:09.795076 (+ 924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:13:09.796099 (+ 1023us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:13:09.798058 (+ 1959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:13:09.798550 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:13:09.798704 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:13:09.799275 (+ 571us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 20:13:09.799495 (+ 220us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0923 20:13:09.799927 (+ 432us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:52694: token expired
Metrics: {"server-negotiator.queue_time_us":81}
W20260923 20:13:09.800357 18129 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:52694: token expired
W20260923 20:13:09.800995 17731 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.1.170.190:44989)
I20260923 20:13:09.816141 18129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:13:09.801810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52706 (local address 127.1.170.190:44989)
0923 20:13:09.802043 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0923 20:13:09.802052 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 20:13:09.802070 (+ 18us) server_negotiation.cc:407] Connection header received
0923 20:13:09.802131 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:13:09.802135 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:13:09.802209 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:13:09.802327 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:13:09.803703 (+ 1376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:13:09.804565 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:13:09.806034 (+ 1469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:13:09.806527 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:13:09.806659 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:13:09.810502 (+ 3843us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:13:09.810507 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:13:09.810510 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:13:09.810550 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:13:09.813203 (+ 2653us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:13:09.814093 (+ 890us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:13:09.814097 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:13:09.814100 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:13:09.814231 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:13:09.814624 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:13:09.814629 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:13:09.814631 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:13:09.814945 (+ 314us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:13:09.815213 (+ 268us) server_negotiation.cc:1035] Waiting for connection context
0923 20:13:09.815555 (+ 342us) server_negotiation.cc:299] Negotiation successful
0923 20:13:09.815794 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103}
I20260923 20:13:09.819692 17810 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52706:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 20:13:09.827250 17810 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 20:13:09.855353 18270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:13:09.841810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41556 (local address 127.1.170.129:43223)
0923 20:13:09.842716 (+ 906us) server_negotiation.cc:206] Beginning negotiation
0923 20:13:09.842725 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 20:13:09.844206 (+ 1481us) server_negotiation.cc:407] Connection header received
0923 20:13:09.844552 (+ 346us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:13:09.844562 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:13:09.844837 (+ 275us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:13:09.845025 (+ 188us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:13:09.846691 (+ 1666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:13:09.848293 (+ 1602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:13:09.852530 (+ 4237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:13:09.853694 (+ 1164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:13:09.853867 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:13:09.854142 (+ 275us) server_negotiation.cc:1035] Waiting for connection context
0923 20:13:09.854835 (+ 693us) server_negotiation.cc:299] Negotiation successful
0923 20:13:09.855018 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":731,"thread_start_us":543,"threads_started":1}
I20260923 20:13:09.855736 18272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:13:09.841717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.129:43223 (local address 127.0.0.1:41556)
0923 20:13:09.844035 (+ 2318us) negotiation.cc:107] Waiting for socket to connect
0923 20:13:09.844091 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0923 20:13:09.844292 (+ 201us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:13:09.845293 (+ 1001us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:13:09.845301 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:13:09.845335 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:13:09.846488 (+ 1153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:13:09.846501 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:13:09.848531 (+ 2030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:13:09.848538 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:13:09.852297 (+ 3759us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:13:09.852305 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:13:09.854300 (+ 1995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:13:09.854306 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:13:09.854527 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:13:09.854580 (+ 53us) client_negotiation.cc:769] Sending connection context
0923 20:13:09.854764 (+ 184us) client_negotiation.cc:240] Negotiation successful
0923 20:13:09.854979 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2108,"mutex_wait_us":205,"thread_start_us":1887,"threads_started":1}
I20260923 20:13:09.855978 18271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:13:09.842282 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55978 (local address 127.1.170.130:35485)
0923 20:13:09.843416 (+ 1134us) server_negotiation.cc:206] Beginning negotiation
0923 20:13:09.843421 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 20:13:09.844528 (+ 1107us) server_negotiation.cc:407] Connection header received
0923 20:13:09.844632 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:13:09.844638 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:13:09.844836 (+ 198us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:13:09.845026 (+ 190us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:13:09.846612 (+ 1586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:13:09.848293 (+ 1681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:13:09.852570 (+ 4277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:13:09.854164 (+ 1594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:13:09.854298 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:13:09.854486 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0923 20:13:09.855519 (+ 1033us) server_negotiation.cc:299] Negotiation successful
0923 20:13:09.855688 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":737,"thread_start_us":686,"threads_started":1}
I20260923 20:13:09.856671 18269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:13:09.841677 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.130:35485 (local address 127.0.0.1:55978)
0923 20:13:09.844035 (+ 2358us) negotiation.cc:107] Waiting for socket to connect
0923 20:13:09.844091 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0923 20:13:09.844292 (+ 201us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:13:09.845293 (+ 1001us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:13:09.845301 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:13:09.845754 (+ 453us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:13:09.846163 (+ 409us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:13:09.846176 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:13:09.848494 (+ 2318us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:13:09.848501 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:13:09.852288 (+ 3787us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:13:09.852304 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:13:09.855070 (+ 2766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:13:09.855077 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:13:09.855287 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:13:09.855337 (+ 50us) client_negotiation.cc:769] Sending connection context
0923 20:13:09.856197 (+ 860us) client_negotiation.cc:240] Negotiation successful
0923 20:13:09.856327 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2047,"mutex_wait_us":30,"thread_start_us":1888,"threads_started":1}
I20260923 20:13:09.857688 18273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:13:09.844856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.131:34241 (local address 127.0.0.1:56206)
0923 20:13:09.849925 (+ 5069us) negotiation.cc:107] Waiting for socket to connect
0923 20:13:09.849955 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0923 20:13:09.850128 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:13:09.850750 (+ 622us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:13:09.850754 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:13:09.850780 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:13:09.851304 (+ 524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:13:09.851314 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:13:09.852937 (+ 1623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:13:09.852942 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:13:09.855070 (+ 2128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:13:09.855083 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:13:09.856859 (+ 1776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:13:09.856866 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:13:09.857067 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:13:09.857109 (+ 42us) client_negotiation.cc:769] Sending connection context
0923 20:13:09.857193 (+ 84us) client_negotiation.cc:240] Negotiation successful
0923 20:13:09.857321 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4848,"mutex_wait_us":49,"thread_start_us":4677,"threads_started":1}
I20260923 20:13:09.858703 18274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:13:09.845405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56206 (local address 127.1.170.131:34241)
0923 20:13:09.849711 (+ 4306us) server_negotiation.cc:206] Beginning negotiation
0923 20:13:09.849720 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 20:13:09.850070 (+ 350us) server_negotiation.cc:407] Connection header received
0923 20:13:09.850312 (+ 242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:13:09.850320 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:13:09.850550 (+ 230us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:13:09.850762 (+ 212us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:13:09.851502 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:13:09.852745 (+ 1243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:13:09.855337 (+ 2592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:13:09.856625 (+ 1288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:13:09.856793 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:13:09.858134 (+ 1341us) server_negotiation.cc:1035] Waiting for connection context
0923 20:13:09.858207 (+ 73us) server_negotiation.cc:299] Negotiation successful
0923 20:13:09.858377 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4091,"thread_start_us":3566,"threads_started":1}
I20260923 20:13:09.860256 17930 tablet_service.cc:1467] Processing CreateTablet for tablet c6bf0a83f20343afa47ead1c6e7dfd8d (DEFAULT_TABLE table=test-table [id=fc66ecb66e0c4d8691c0b466ae61d633]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 20:13:09.861109 18062 tablet_service.cc:1467] Processing CreateTablet for tablet c6bf0a83f20343afa47ead1c6e7dfd8d (DEFAULT_TABLE table=test-table [id=fc66ecb66e0c4d8691c0b466ae61d633]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 20:13:09.861922 17930 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c6bf0a83f20343afa47ead1c6e7dfd8d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:13:09.862399 18062 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c6bf0a83f20343afa47ead1c6e7dfd8d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:13:09.862244 18194 tablet_service.cc:1467] Processing CreateTablet for tablet c6bf0a83f20343afa47ead1c6e7dfd8d (DEFAULT_TABLE table=test-table [id=fc66ecb66e0c4d8691c0b466ae61d633]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 20:13:09.863560 18194 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c6bf0a83f20343afa47ead1c6e7dfd8d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:13:09.875953 18275 tablet_bootstrap.cc:492] T c6bf0a83f20343afa47ead1c6e7dfd8d P 908ce75423b74581b7328602946ef436: Bootstrap starting.
I20260923 20:13:09.878473 18277 tablet_bootstrap.cc:492] T c6bf0a83f20343afa47ead1c6e7dfd8d P b5dae6161ce245f8b7586278900d0d1e: Bootstrap starting.
I20260923 20:13:09.880080 18275 tablet_bootstrap.cc:654] T c6bf0a83f20343afa47ead1c6e7dfd8d P 908ce75423b74581b7328602946ef436: Neither blocks nor log segments found. Creating new log.
I20260923 20:13:09.880620 18276 tablet_bootstrap.cc:492] T c6bf0a83f20343afa47ead1c6e7dfd8d P da1071f8ba4c4d93b8a8d1d6f143babe: Bootstrap starting.
I20260923 20:13:09.881649 18275 log.cc:824] T c6bf0a83f20343afa47ead1c6e7dfd8d P 908ce75423b74581b7328602946ef436: Log is configured to *not* fsync() on all Append() calls
I20260923 20:13:09.882246 18277 tablet_bootstrap.cc:654] T c6bf0a83f20343afa47ead1c6e7dfd8d P b5dae6161ce245f8b7586278900d0d1e: Neither blocks nor log segments found. Creating new log.
I20260923 20:13:09.883898 18277 log.cc:824] T c6bf0a83f20343afa47ead1c6e7dfd8d P b5dae6161ce245f8b7586278900d0d1e: Log is configured to *not* fsync() on all Append() calls
I20260923 20:13:09.884192 18276 tablet_bootstrap.cc:654] T c6bf0a83f20343afa47ead1c6e7dfd8d P da1071f8ba4c4d93b8a8d1d6f143babe: Neither blocks nor log segments found. Creating new log.
I20260923 20:13:09.885535 18276 log.cc:824] T c6bf0a83f20343afa47ead1c6e7dfd8d P da1071f8ba4c4d93b8a8d1d6f143babe: Log is configured to *not* fsync() on all Append() calls
I20260923 20:13:09.887490 18275 tablet_bootstrap.cc:492] T c6bf0a83f20343afa47ead1c6e7dfd8d P 908ce75423b74581b7328602946ef436: No bootstrap required, opened a new log
I20260923 20:13:09.887490 18277 tablet_bootstrap.cc:492] T c6bf0a83f20343afa47ead1c6e7dfd8d P b5dae6161ce245f8b7586278900d0d1e: No bootstrap required, opened a new log
I20260923 20:13:09.887789 18275 ts_tablet_manager.cc:1397] T c6bf0a83f20343afa47ead1c6e7dfd8d P 908ce75423b74581b7328602946ef436: Time spent bootstrapping tablet: real 0.012s user 0.005s sys 0.005s
I20260923 20:13:09.887789 18277 ts_tablet_manager.cc:1397] T c6bf0a83f20343afa47ead1c6e7dfd8d P b5dae6161ce245f8b7586278900d0d1e: Time spent bootstrapping tablet: real 0.010s user 0.001s sys 0.007s
I20260923 20:13:09.888211 18276 tablet_bootstrap.cc:492] T c6bf0a83f20343afa47ead1c6e7dfd8d P da1071f8ba4c4d93b8a8d1d6f143babe: No bootstrap required, opened a new log
I20260923 20:13:09.888484 18276 ts_tablet_manager.cc:1397] T c6bf0a83f20343afa47ead1c6e7dfd8d P da1071f8ba4c4d93b8a8d1d6f143babe: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260923 20:13:09.897369 18276 raft_consensus.cc:348] T c6bf0a83f20343afa47ead1c6e7dfd8d P da1071f8ba4c4d93b8a8d1d6f143babe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "908ce75423b74581b7328602946ef436" member_type: VOTER last_known_addr { host: "127.1.170.130" port: 35485 } } peers { permanent_uuid: "da1071f8ba4c4d93b8a8d1d6f143babe" member_type: VOTER last_known_addr { host: "127.1.170.129" port: 43223 } } peers { permanent_uuid: "b5dae6161ce245f8b7586278900d0d1e" member_type: VOTER last_known_addr { host: "127.1.170.131" port: 34241 } }
I20260923 20:13:09.897830 18276 raft_consensus.cc:374] T c6bf0a83f20343afa47ead1c6e7dfd8d P da1071f8ba4c4d93b8a8d1d6f143babe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:13:09.897964 18276 raft_consensus.cc:729] T c6bf0a83f20343afa47ead1c6e7dfd8d P da1071f8ba4c4d93b8a8d1d6f143babe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: da1071f8ba4c4d93b8a8d1d6f143babe, State: Initialized, Role: FOLLOWER
I20260923 20:13:09.898645 18276 consensus_queue.cc:260] T c6bf0a83f20343afa47ead1c6e7dfd8d P da1071f8ba4c4d93b8a8d1d6f143babe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "908ce75423b74581b7328602946ef436" member_type: VOTER last_known_addr { host: "127.1.170.130" port: 35485 } } peers { permanent_uuid: "da1071f8ba4c4d93b8a8d1d6f143babe" member_type: VOTER last_known_addr { host: "127.1.170.129" port: 43223 } } peers { permanent_uuid: "b5dae6161ce245f8b7586278900d0d1e" member_type: VOTER last_known_addr { host: "127.1.170.131" port: 34241 } }
I20260923 20:13:09.899037 18277 raft_consensus.cc:348] T c6bf0a83f20343afa47ead1c6e7dfd8d P b5dae6161ce245f8b7586278900d0d1e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "908ce75423b74581b7328602946ef436" member_type: VOTER last_known_addr { host: "127.1.170.130" port: 35485 } } peers { permanent_uuid: "da1071f8ba4c4d93b8a8d1d6f143babe" member_type: VOTER last_known_addr { host: "127.1.170.129" port: 43223 } } peers { permanent_uuid: "b5dae6161ce245f8b7586278900d0d1e" member_type: VOTER last_known_addr { host: "127.1.170.131" port: 34241 } }
I20260923 20:13:09.899037 18275 raft_consensus.cc:348] T c6bf0a83f20343afa47ead1c6e7dfd8d P 908ce75423b74581b7328602946ef436 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "908ce75423b74581b7328602946ef436" member_type: VOTER last_known_addr { host: "127.1.170.130" port: 35485 } } peers { permanent_uuid: "da1071f8ba4c4d93b8a8d1d6f143babe" member_type: VOTER last_known_addr { host: "127.1.170.129" port: 43223 } } peers { permanent_uuid: "b5dae6161ce245f8b7586278900d0d1e" member_type: VOTER last_known_addr { host: "127.1.170.131" port: 34241 } }
I20260923 20:13:09.899480 18275 raft_consensus.cc:374] T c6bf0a83f20343afa47ead1c6e7dfd8d P 908ce75423b74581b7328602946ef436 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:13:09.899480 18277 raft_consensus.cc:374] T c6bf0a83f20343afa47ead1c6e7dfd8d P b5dae6161ce245f8b7586278900d0d1e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:13:09.899585 18277 raft_consensus.cc:729] T c6bf0a83f20343afa47ead1c6e7dfd8d P b5dae6161ce245f8b7586278900d0d1e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b5dae6161ce245f8b7586278900d0d1e, State: Initialized, Role: FOLLOWER
I20260923 20:13:09.899585 18275 raft_consensus.cc:729] T c6bf0a83f20343afa47ead1c6e7dfd8d P 908ce75423b74581b7328602946ef436 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 908ce75423b74581b7328602946ef436, State: Initialized, Role: FOLLOWER
I20260923 20:13:09.900315 18277 consensus_queue.cc:260] T c6bf0a83f20343afa47ead1c6e7dfd8d P b5dae6161ce245f8b7586278900d0d1e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "908ce75423b74581b7328602946ef436" member_type: VOTER last_known_addr { host: "127.1.170.130" port: 35485 } } peers { permanent_uuid: "da1071f8ba4c4d93b8a8d1d6f143babe" member_type: VOTER last_known_addr { host: "127.1.170.129" port: 43223 } } peers { permanent_uuid: "b5dae6161ce245f8b7586278900d0d1e" member_type: VOTER last_known_addr { host: "127.1.170.131" port: 34241 } }
I20260923 20:13:09.900315 18275 consensus_queue.cc:260] T c6bf0a83f20343afa47ead1c6e7dfd8d P 908ce75423b74581b7328602946ef436 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "908ce75423b74581b7328602946ef436" member_type: VOTER last_known_addr { host: "127.1.170.130" port: 35485 } } peers { permanent_uuid: "da1071f8ba4c4d93b8a8d1d6f143babe" member_type: VOTER last_known_addr { host: "127.1.170.129" port: 43223 } } peers { permanent_uuid: "b5dae6161ce245f8b7586278900d0d1e" member_type: VOTER last_known_addr { host: "127.1.170.131" port: 34241 } }
I20260923 20:13:09.900738 18276 ts_tablet_manager.cc:1428] T c6bf0a83f20343afa47ead1c6e7dfd8d P da1071f8ba4c4d93b8a8d1d6f143babe: Time spent starting tablet: real 0.012s user 0.012s sys 0.000s
I20260923 20:13:09.901856 18259 heartbeater.cc:503] Master 127.1.170.190:44989 was elected leader, sending a full tablet report...
I20260923 20:13:09.902743 18277 ts_tablet_manager.cc:1428] T c6bf0a83f20343afa47ead1c6e7dfd8d P b5dae6161ce245f8b7586278900d0d1e: Time spent starting tablet: real 0.015s user 0.008s sys 0.008s
I20260923 20:13:09.905519 18127 heartbeater.cc:503] Master 127.1.170.190:44989 was elected leader, sending a full tablet report...
I20260923 20:13:09.905853 18275 ts_tablet_manager.cc:1428] T c6bf0a83f20343afa47ead1c6e7dfd8d P 908ce75423b74581b7328602946ef436: Time spent starting tablet: real 0.018s user 0.011s sys 0.007s
W20260923 20:13:09.979650 17996 tablet.cc:2369] T c6bf0a83f20343afa47ead1c6e7dfd8d P da1071f8ba4c4d93b8a8d1d6f143babe: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260923 20:13:09.999965 18260 tablet.cc:2369] T c6bf0a83f20343afa47ead1c6e7dfd8d P b5dae6161ce245f8b7586278900d0d1e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260923 20:13:10.010226 18128 tablet.cc:2369] T c6bf0a83f20343afa47ead1c6e7dfd8d P 908ce75423b74581b7328602946ef436: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 20:13:10.102475 18282 raft_consensus.cc:482] T c6bf0a83f20343afa47ead1c6e7dfd8d P b5dae6161ce245f8b7586278900d0d1e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 20:13:10.102811 18282 raft_consensus.cc:504] T c6bf0a83f20343afa47ead1c6e7dfd8d P b5dae6161ce245f8b7586278900d0d1e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "908ce75423b74581b7328602946ef436" member_type: VOTER last_known_addr { host: "127.1.170.130" port: 35485 } } peers { permanent_uuid: "da1071f8ba4c4d93b8a8d1d6f143babe" member_type: VOTER last_known_addr { host: "127.1.170.129" port: 43223 } } peers { permanent_uuid: "b5dae6161ce245f8b7586278900d0d1e" member_type: VOTER last_known_addr { host: "127.1.170.131" port: 34241 } }
I20260923 20:13:10.104641 18282 leader_election.cc:288] T c6bf0a83f20343afa47ead1c6e7dfd8d P b5dae6161ce245f8b7586278900d0d1e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 908ce75423b74581b7328602946ef436 (127.1.170.130:35485), da1071f8ba4c4d93b8a8d1d6f143babe (127.1.170.129:43223)
I20260923 20:13:10.111203 18261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:13:10.105060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.130:35485 (local address 127.1.170.131:49571)
0923 20:13:10.105405 (+ 345us) negotiation.cc:107] Waiting for socket to connect
0923 20:13:10.105433 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0923 20:13:10.105617 (+ 184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:13:10.106038 (+ 421us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:13:10.106051 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:13:10.106080 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:13:10.106439 (+ 359us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:13:10.106454 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:13:10.107835 (+ 1381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:13:10.107841 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:13:10.109230 (+ 1389us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:13:10.109242 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:13:10.110423 (+ 1181us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:13:10.110429 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:13:10.110607 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:13:10.110630 (+ 23us) client_negotiation.cc:769] Sending connection context
0923 20:13:10.110743 (+ 113us) client_negotiation.cc:240] Negotiation successful
0923 20:13:10.110883 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":56,"mutex_wait_us":3}
I20260923 20:13:10.111182 18271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:13:10.105410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.170.131:49571 (local address 127.1.170.130:35485)
0923 20:13:10.105596 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0923 20:13:10.105601 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 20:13:10.105621 (+ 20us) server_negotiation.cc:407] Connection header received
0923 20:13:10.105763 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:13:10.105767 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:13:10.105827 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:13:10.106016 (+ 189us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:13:10.106605 (+ 589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:13:10.107358 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:13:10.109398 (+ 2040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:13:10.110260 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:13:10.110426 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:13:10.110511 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0923 20:13:10.110768 (+ 257us) server_negotiation.cc:299] Negotiation successful
0923 20:13:10.110894 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260923 20:13:10.111984 18270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:13:10.105371 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.170.131:55387 (local address 127.1.170.129:43223)
0923 20:13:10.105602 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0923 20:13:10.105608 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:13:10.106628 (+ 1020us) server_negotiation.cc:407] Connection header received
0923 20:13:10.106804 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:13:10.106812 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:13:10.106899 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:13:10.107001 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:13:10.107557 (+ 556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:13:10.108446 (+ 889us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:13:10.110198 (+ 1752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:13:10.110967 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:13:10.111166 (+ 199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:13:10.111237 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0923 20:13:10.111599 (+ 362us) server_negotiation.cc:299] Negotiation successful
0923 20:13:10.111743 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260923 20:13:10.112582 18284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:13:10.105060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.129:43223 (local address 127.1.170.131:55387)
0923 20:13:10.106470 (+ 1410us) negotiation.cc:107] Waiting for socket to connect
0923 20:13:10.106527 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0923 20:13:10.106655 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:13:10.107014 (+ 359us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:13:10.107018 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:13:10.107041 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:13:10.107410 (+ 369us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:13:10.107420 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:13:10.108639 (+ 1219us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:13:10.108645 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:13:10.110028 (+ 1383us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:13:10.110040 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:13:10.111181 (+ 1141us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:13:10.111191 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:13:10.111390 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:13:10.111437 (+ 47us) client_negotiation.cc:769] Sending connection context
0923 20:13:10.112170 (+ 733us) client_negotiation.cc:240] Negotiation successful
0923 20:13:10.112314 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1308,"mutex_wait_us":570,"thread_start_us":453,"threads_started":1}
I20260923 20:13:10.112823 18082 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6bf0a83f20343afa47ead1c6e7dfd8d" candidate_uuid: "b5dae6161ce245f8b7586278900d0d1e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "908ce75423b74581b7328602946ef436" is_pre_election: true
I20260923 20:13:10.113415 18082 raft_consensus.cc:2454] T c6bf0a83f20343afa47ead1c6e7dfd8d P 908ce75423b74581b7328602946ef436 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b5dae6161ce245f8b7586278900d0d1e in term 0.
I20260923 20:13:10.113584 17950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6bf0a83f20343afa47ead1c6e7dfd8d" candidate_uuid: "b5dae6161ce245f8b7586278900d0d1e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "da1071f8ba4c4d93b8a8d1d6f143babe" is_pre_election: true
I20260923 20:13:10.114097 17950 raft_consensus.cc:2454] T c6bf0a83f20343afa47ead1c6e7dfd8d P da1071f8ba4c4d93b8a8d1d6f143babe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b5dae6161ce245f8b7586278900d0d1e in term 0.
I20260923 20:13:10.114125 18150 leader_election.cc:302] T c6bf0a83f20343afa47ead1c6e7dfd8d P b5dae6161ce245f8b7586278900d0d1e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 908ce75423b74581b7328602946ef436, b5dae6161ce245f8b7586278900d0d1e; no voters:
I20260923 20:13:10.114518 18282 raft_consensus.cc:2781] T c6bf0a83f20343afa47ead1c6e7dfd8d P b5dae6161ce245f8b7586278900d0d1e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 20:13:10.114660 18282 raft_consensus.cc:482] T c6bf0a83f20343afa47ead1c6e7dfd8d P b5dae6161ce245f8b7586278900d0d1e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 20:13:10.114780 18282 raft_consensus.cc:3037] T c6bf0a83f20343afa47ead1c6e7dfd8d P b5dae6161ce245f8b7586278900d0d1e [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:13:10.116961 18282 raft_consensus.cc:504] T c6bf0a83f20343afa47ead1c6e7dfd8d P b5dae6161ce245f8b7586278900d0d1e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "908ce75423b74581b7328602946ef436" member_type: VOTER last_known_addr { host: "127.1.170.130" port: 35485 } } peers { permanent_uuid: "da1071f8ba4c4d93b8a8d1d6f143babe" member_type: VOTER last_known_addr { host: "127.1.170.129" port: 43223 } } peers { permanent_uuid: "b5dae6161ce245f8b7586278900d0d1e" member_type: VOTER last_known_addr { host: "127.1.170.131" port: 34241 } }
I20260923 20:13:10.117667 18282 leader_election.cc:288] T c6bf0a83f20343afa47ead1c6e7dfd8d P b5dae6161ce245f8b7586278900d0d1e [CANDIDATE]: Term 1 election: Requested vote from peers 908ce75423b74581b7328602946ef436 (127.1.170.130:35485), da1071f8ba4c4d93b8a8d1d6f143babe (127.1.170.129:43223)
I20260923 20:13:10.118207 18082 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6bf0a83f20343afa47ead1c6e7dfd8d" candidate_uuid: "b5dae6161ce245f8b7586278900d0d1e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "908ce75423b74581b7328602946ef436"
I20260923 20:13:10.118274 17950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6bf0a83f20343afa47ead1c6e7dfd8d" candidate_uuid: "b5dae6161ce245f8b7586278900d0d1e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "da1071f8ba4c4d93b8a8d1d6f143babe"
I20260923 20:13:10.118453 18082 raft_consensus.cc:3037] T c6bf0a83f20343afa47ead1c6e7dfd8d P 908ce75423b74581b7328602946ef436 [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:13:10.118505 17950 raft_consensus.cc:3037] T c6bf0a83f20343afa47ead1c6e7dfd8d P da1071f8ba4c4d93b8a8d1d6f143babe [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:13:10.120718 18082 raft_consensus.cc:2454] T c6bf0a83f20343afa47ead1c6e7dfd8d P 908ce75423b74581b7328602946ef436 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b5dae6161ce245f8b7586278900d0d1e in term 1.
I20260923 20:13:10.121037 17950 raft_consensus.cc:2454] T c6bf0a83f20343afa47ead1c6e7dfd8d P da1071f8ba4c4d93b8a8d1d6f143babe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b5dae6161ce245f8b7586278900d0d1e in term 1.
I20260923 20:13:10.121248 18150 leader_election.cc:302] T c6bf0a83f20343afa47ead1c6e7dfd8d P b5dae6161ce245f8b7586278900d0d1e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 908ce75423b74581b7328602946ef436, b5dae6161ce245f8b7586278900d0d1e; no voters:
I20260923 20:13:10.121618 18282 raft_consensus.cc:2781] T c6bf0a83f20343afa47ead1c6e7dfd8d P b5dae6161ce245f8b7586278900d0d1e [term 1 FOLLOWER]: Leader election won for term 1
I20260923 20:13:10.122174 18282 raft_consensus.cc:686] T c6bf0a83f20343afa47ead1c6e7dfd8d P b5dae6161ce245f8b7586278900d0d1e [term 1 LEADER]: Becoming Leader. State: Replica: b5dae6161ce245f8b7586278900d0d1e, State: Running, Role: LEADER
I20260923 20:13:10.122745 18282 consensus_queue.cc:237] T c6bf0a83f20343afa47ead1c6e7dfd8d P b5dae6161ce245f8b7586278900d0d1e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "908ce75423b74581b7328602946ef436" member_type: VOTER last_known_addr { host: "127.1.170.130" port: 35485 } } peers { permanent_uuid: "da1071f8ba4c4d93b8a8d1d6f143babe" member_type: VOTER last_known_addr { host: "127.1.170.129" port: 43223 } } peers { permanent_uuid: "b5dae6161ce245f8b7586278900d0d1e" member_type: VOTER last_known_addr { host: "127.1.170.131" port: 34241 } }
I20260923 20:13:10.127908 17809 catalog_manager.cc:5495] T c6bf0a83f20343afa47ead1c6e7dfd8d P b5dae6161ce245f8b7586278900d0d1e reported cstate change: term changed from 0 to 1, leader changed from <none> to b5dae6161ce245f8b7586278900d0d1e (127.1.170.131). New cstate: current_term: 1 leader_uuid: "b5dae6161ce245f8b7586278900d0d1e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "908ce75423b74581b7328602946ef436" member_type: VOTER last_known_addr { host: "127.1.170.130" port: 35485 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "da1071f8ba4c4d93b8a8d1d6f143babe" member_type: VOTER last_known_addr { host: "127.1.170.129" port: 43223 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b5dae6161ce245f8b7586278900d0d1e" member_type: VOTER last_known_addr { host: "127.1.170.131" port: 34241 } health_report { overall_health: HEALTHY } } }
I20260923 20:13:10.159251 18274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:13:10.153549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56216 (local address 127.1.170.131:34241)
0923 20:13:10.153740 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0923 20:13:10.153745 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 20:13:10.153763 (+ 18us) server_negotiation.cc:407] Connection header received
0923 20:13:10.154017 (+ 254us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:13:10.154020 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:13:10.154116 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:13:10.154250 (+ 134us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 20:13:10.154627 (+ 377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:13:10.155391 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:13:10.156868 (+ 1477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:13:10.157275 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:13:10.157395 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:13:10.158028 (+ 633us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 20:13:10.158564 (+ 536us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 20:13:10.158674 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0923 20:13:10.158858 (+ 184us) server_negotiation.cc:299] Negotiation successful
0923 20:13:10.158938 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260923 20:13:10.168491 18082 raft_consensus.cc:1261] T c6bf0a83f20343afa47ead1c6e7dfd8d P 908ce75423b74581b7328602946ef436 [term 1 FOLLOWER]: Refusing update from remote peer b5dae6161ce245f8b7586278900d0d1e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 20:13:10.168491 17950 raft_consensus.cc:1261] T c6bf0a83f20343afa47ead1c6e7dfd8d P da1071f8ba4c4d93b8a8d1d6f143babe [term 1 FOLLOWER]: Refusing update from remote peer b5dae6161ce245f8b7586278900d0d1e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 20:13:10.169520 18285 consensus_queue.cc:1035] T c6bf0a83f20343afa47ead1c6e7dfd8d P b5dae6161ce245f8b7586278900d0d1e [LEADER]: Connected to new peer: Peer: permanent_uuid: "908ce75423b74581b7328602946ef436" member_type: VOTER last_known_addr { host: "127.1.170.130" port: 35485 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 20:13:10.169904 18282 consensus_queue.cc:1035] T c6bf0a83f20343afa47ead1c6e7dfd8d P b5dae6161ce245f8b7586278900d0d1e [LEADER]: Connected to new peer: Peer: permanent_uuid: "da1071f8ba4c4d93b8a8d1d6f143babe" member_type: VOTER last_known_addr { host: "127.1.170.129" port: 43223 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 20:13:10.196517 17809 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52706:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 20:13:10.196807 17809 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52706:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 20:13:10.200837 17809 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5cef025810a84435a7e1983ca1e0ac5b: Sending DeleteTablet for 3 replicas of tablet c6bf0a83f20343afa47ead1c6e7dfd8d
I20260923 20:13:10.202127 18062 tablet_service.cc:1514] Processing DeleteTablet for tablet c6bf0a83f20343afa47ead1c6e7dfd8d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 20:13:10 UTC) from {username='kudu', principal='kudu/127.1.170.190@KRBTEST.COM'} at 127.0.0.1:55978
I20260923 20:13:10.202257 18194 tablet_service.cc:1514] Processing DeleteTablet for tablet c6bf0a83f20343afa47ead1c6e7dfd8d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 20:13:10 UTC) from {username='kudu', principal='kudu/127.1.170.190@KRBTEST.COM'} at 127.0.0.1:56206
I20260923 20:13:10.202440 1706 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskqUuku6/build/asan/bin/kudu pid 17847
I20260923 20:13:10.202621 17930 tablet_service.cc:1514] Processing DeleteTablet for tablet c6bf0a83f20343afa47ead1c6e7dfd8d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 20:13:10 UTC) from {username='kudu', principal='kudu/127.1.170.190@KRBTEST.COM'} at 127.0.0.1:41556
I20260923 20:13:10.204977 18298 tablet_replica.cc:330] T c6bf0a83f20343afa47ead1c6e7dfd8d P b5dae6161ce245f8b7586278900d0d1e: stopping tablet replica
I20260923 20:13:10.205555 18298 raft_consensus.cc:2229] T c6bf0a83f20343afa47ead1c6e7dfd8d P b5dae6161ce245f8b7586278900d0d1e [term 1 LEADER]: Raft consensus shutting down.
I20260923 20:13:10.206777 18299 tablet_replica.cc:330] T c6bf0a83f20343afa47ead1c6e7dfd8d P da1071f8ba4c4d93b8a8d1d6f143babe: stopping tablet replica
I20260923 20:13:10.207389 18299 raft_consensus.cc:2229] T c6bf0a83f20343afa47ead1c6e7dfd8d P da1071f8ba4c4d93b8a8d1d6f143babe [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 20:13:10.208045 18299 raft_consensus.cc:2258] T c6bf0a83f20343afa47ead1c6e7dfd8d P da1071f8ba4c4d93b8a8d1d6f143babe [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 20:13:10.208755 18298 raft_consensus.cc:2258] T c6bf0a83f20343afa47ead1c6e7dfd8d P b5dae6161ce245f8b7586278900d0d1e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 20:13:10.223099 18270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:13:10.203646 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41564 (local address 127.1.170.129:43223)
0923 20:13:10.203870 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0923 20:13:10.203879 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 20:13:10.203905 (+ 26us) server_negotiation.cc:407] Connection header received
0923 20:13:10.203979 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:13:10.203987 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:13:10.204101 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:13:10.204226 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:13:10.206636 (+ 2410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:13:10.207848 (+ 1212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:13:10.210082 (+ 2234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:13:10.210763 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:13:10.210884 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:13:10.214676 (+ 3792us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:13:10.214695 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:13:10.214714 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:13:10.214782 (+ 68us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:13:10.218709 (+ 3927us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:13:10.219720 (+ 1011us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:13:10.219732 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:13:10.219737 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:13:10.219934 (+ 197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:13:10.220392 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:13:10.220397 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:13:10.220400 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:13:10.220886 (+ 486us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:13:10.221193 (+ 307us) server_negotiation.cc:1035] Waiting for connection context
0923 20:13:10.222449 (+ 1256us) server_negotiation.cc:299] Negotiation successful
0923 20:13:10.222691 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260923 20:13:10.223417 18298 ts_tablet_manager.cc:1905] T c6bf0a83f20343afa47ead1c6e7dfd8d P b5dae6161ce245f8b7586278900d0d1e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 20:13:10.223796 17990 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:13:10.226487 18297 tablet_replica.cc:330] T c6bf0a83f20343afa47ead1c6e7dfd8d P 908ce75423b74581b7328602946ef436: stopping tablet replica
I20260923 20:13:10.226977 18297 raft_consensus.cc:2229] T c6bf0a83f20343afa47ead1c6e7dfd8d P 908ce75423b74581b7328602946ef436 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 20:13:10.227511 18297 raft_consensus.cc:2258] T c6bf0a83f20343afa47ead1c6e7dfd8d P 908ce75423b74581b7328602946ef436 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 20:13:10.229935 18297 ts_tablet_manager.cc:1905] T c6bf0a83f20343afa47ead1c6e7dfd8d P 908ce75423b74581b7328602946ef436: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 20:13:10.245630 18297 ts_tablet_manager.cc:1918] T c6bf0a83f20343afa47ead1c6e7dfd8d P 908ce75423b74581b7328602946ef436: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 20:13:10.245829 18297 log.cc:1196] T c6bf0a83f20343afa47ead1c6e7dfd8d P 908ce75423b74581b7328602946ef436: Deleting WAL directory at /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194121577027-1706-0/minicluster-data/ts-1/wal/wals/c6bf0a83f20343afa47ead1c6e7dfd8d
I20260923 20:13:10.246426 18297 ts_tablet_manager.cc:1939] T c6bf0a83f20343afa47ead1c6e7dfd8d P 908ce75423b74581b7328602946ef436: Deleting consensus metadata
I20260923 20:13:10.248394 17797 catalog_manager.cc:4841] TS 908ce75423b74581b7328602946ef436 (127.1.170.130:35485): tablet c6bf0a83f20343afa47ead1c6e7dfd8d (table test-table [id=fc66ecb66e0c4d8691c0b466ae61d633]) successfully deleted
I20260923 20:13:10.248505 18298 ts_tablet_manager.cc:1918] T c6bf0a83f20343afa47ead1c6e7dfd8d P b5dae6161ce245f8b7586278900d0d1e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 20:13:10.248724 18298 log.cc:1196] T c6bf0a83f20343afa47ead1c6e7dfd8d P b5dae6161ce245f8b7586278900d0d1e: Deleting WAL directory at /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194121577027-1706-0/minicluster-data/ts-2/wal/wals/c6bf0a83f20343afa47ead1c6e7dfd8d
I20260923 20:13:10.249202 18298 ts_tablet_manager.cc:1939] T c6bf0a83f20343afa47ead1c6e7dfd8d P b5dae6161ce245f8b7586278900d0d1e: Deleting consensus metadata
I20260923 20:13:10.250820 17794 catalog_manager.cc:4841] TS b5dae6161ce245f8b7586278900d0d1e (127.1.170.131:34241): tablet c6bf0a83f20343afa47ead1c6e7dfd8d (table test-table [id=fc66ecb66e0c4d8691c0b466ae61d633]) successfully deleted
I20260923 20:13:10.228157 18299 ts_tablet_manager.cc:1905] T c6bf0a83f20343afa47ead1c6e7dfd8d P da1071f8ba4c4d93b8a8d1d6f143babe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 20:13:10.364693 1706 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqUuku6/build/asan/bin/kudu with pid 17847
I20260923 20:13:10.384320 1706 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskqUuku6/build/asan/bin/kudu pid 17999
W20260923 20:13:10.384352 17794 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.1.170.129:43223 (error 108)
I20260923 20:13:10.387928 18302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:13:10.385774 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.129:43223 (local address 127.0.0.1:41574)
0923 20:13:10.387579 (+ 1805us) negotiation.cc:107] Waiting for socket to connect
0923 20:13:10.387721 (+ 142us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.170.129:43223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1666,"thread_start_us":1567,"threads_started":1}
W20260923 20:13:10.388206 17794 catalog_manager.cc:4584] TS da1071f8ba4c4d93b8a8d1d6f143babe (127.1.170.129:43223): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet c6bf0a83f20343afa47ead1c6e7dfd8d: Network error: Client connection negotiation failed: client connection to 127.1.170.129:43223: connect: Connection refused (error 111)
I20260923 20:13:10.401592 18271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:13:10.385463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55988 (local address 127.1.170.130:35485)
0923 20:13:10.385641 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0923 20:13:10.385646 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 20:13:10.385662 (+ 16us) server_negotiation.cc:407] Connection header received
0923 20:13:10.385715 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:13:10.385719 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:13:10.385778 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:13:10.386130 (+ 352us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:13:10.387254 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:13:10.388324 (+ 1070us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:13:10.390416 (+ 2092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:13:10.390897 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:13:10.391005 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:13:10.394865 (+ 3860us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:13:10.394883 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:13:10.394908 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:13:10.395005 (+ 97us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:13:10.398468 (+ 3463us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:13:10.399431 (+ 963us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:13:10.399438 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:13:10.399441 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:13:10.399578 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:13:10.400018 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:13:10.400022 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:13:10.400024 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:13:10.400427 (+ 403us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:13:10.400666 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0923 20:13:10.401076 (+ 410us) server_negotiation.cc:299] Negotiation successful
0923 20:13:10.401250 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260923 20:13:10.402650 18122 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:13:10.442075 18302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:13:10.441425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.129:43223 (local address 127.0.0.1:41580)
0923 20:13:10.441717 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0923 20:13:10.441845 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.170.129:43223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131}
I20260923 20:13:10.443318 18302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:13:10.442871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.129:43223 (local address 127.0.0.1:41588)
0923 20:13:10.443064 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0923 20:13:10.443156 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.170.129:43223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260923 20:13:10.497852 1706 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqUuku6/build/asan/bin/kudu with pid 17999
I20260923 20:13:10.504611 18302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:13:10.503930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.129:43223 (local address 127.0.0.1:41596)
0923 20:13:10.504247 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0923 20:13:10.504395 (+ 148us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.170.129:43223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117}
I20260923 20:13:10.506227 18302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:13:10.505663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.129:43223 (local address 127.0.0.1:41600)
0923 20:13:10.505952 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0923 20:13:10.506042 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.170.129:43223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":157}
I20260923 20:13:10.520874 1706 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskqUuku6/build/asan/bin/kudu pid 18131
I20260923 20:13:10.537124 18274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:13:10.521853 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56226 (local address 127.1.170.131:34241)
0923 20:13:10.522142 (+ 289us) server_negotiation.cc:206] Beginning negotiation
0923 20:13:10.522148 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:13:10.522164 (+ 16us) server_negotiation.cc:407] Connection header received
0923 20:13:10.522218 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:13:10.522222 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:13:10.522282 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:13:10.522404 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:13:10.523683 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:13:10.524543 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:13:10.525871 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:13:10.526397 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:13:10.526566 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:13:10.529641 (+ 3075us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:13:10.529656 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:13:10.529677 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:13:10.529725 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:13:10.533667 (+ 3942us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:13:10.534730 (+ 1063us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:13:10.534738 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:13:10.534741 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:13:10.534924 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:13:10.535453 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:13:10.535457 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:13:10.535459 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:13:10.535845 (+ 386us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:13:10.536116 (+ 271us) server_negotiation.cc:1035] Waiting for connection context
0923 20:13:10.536534 (+ 418us) server_negotiation.cc:299] Negotiation successful
0923 20:13:10.536722 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124}
I20260923 20:13:10.537974 18254 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:13:10.587256 18302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:13:10.586652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.129:43223 (local address 127.0.0.1:41612)
0923 20:13:10.586941 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0923 20:13:10.587089 (+ 148us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.170.129:43223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133}
I20260923 20:13:10.588486 18302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:13:10.588054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.129:43223 (local address 127.0.0.1:41628)
0923 20:13:10.588253 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0923 20:13:10.588340 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.170.129:43223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96}
I20260923 20:13:10.665520 1706 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqUuku6/build/asan/bin/kudu with pid 18131
I20260923 20:13:10.688167 1706 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskqUuku6/build/asan/bin/kudu pid 17777
I20260923 20:13:10.689559 17840 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:13:10.809794 18302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:13:10.808992 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.129:43223 (local address 127.0.0.1:41642)
0923 20:13:10.809364 (+ 372us) negotiation.cc:107] Waiting for socket to connect
0923 20:13:10.809549 (+ 185us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.170.129:43223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113}
I20260923 20:13:10.809968 1706 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqUuku6/build/asan/bin/kudu with pid 17777
I20260923 20:13:30.834899 1706 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqUuku6/build/asan/bin/kudu
/tmp/dist-test-taskqUuku6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194121577027-1706-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194121577027-1706-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194121577027-1706-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194121577027-1706-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.170.190:44989
--webserver_interface=127.1.170.190
--webserver_port=34517
--keytab_file=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194121577027-1706-0/krb5kdc/kudu_127.1.170.190.keytab
--principal=kudu/127.1.170.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.170.148:42533
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.170.190:44989
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194121577027-1706-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194121577027-1706-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194121577027-1706-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:13:31.193759 18306 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:13:31.194169 18306 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:13:31.194267 18306 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:13:31.201862 18306 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260923 20:13:31.202062 18306 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260923 20:13:31.202102 18306 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260923 20:13:31.202148 18306 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 20:13:31.202196 18306 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:13:31.202267 18306 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:13:31.202314 18306 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 20:13:31.202363 18306 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 20:13:31.213447 18306 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.170.148:42533
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194121577027-1706-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194121577027-1706-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.1.170.190:44989
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194121577027-1706-0/krb5kdc/kudu_127.1.170.190.keytab
--principal=kudu/127.1.170.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.170.190:44989
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194121577027-1706-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.170.190
--webserver_port=34517
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194121577027-1706-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:13:31.214977 18306 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:13:31.216445 18306 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:13:31.229219 18312 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:13:31.229287 18314 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:13:31.229193 18311 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:13:31.229588 18306 server_base.cc:1034] running on GCE node
I20260923 20:13:31.230985 18306 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:13:31.232640 18306 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:13:31.234066 18306 hybrid_clock.cc:648] HybridClock initialized: now 1790194411233961 us; error 91 us; skew 500 ppm
I20260923 20:13:31.234438 18306 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:13:31 dist-test-slave-th4g krb5kdc[17225](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194411, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.170.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:13:31.237984 18306 init.cc:377] Logged in from keytab as kudu/127.1.170.190@KRBTEST.COM (short username kudu)
I20260923 20:13:31.239683 18306 webserver.cc:460] Webserver started at http://127.1.170.190:34517/ using document root <none> and password file <none>
I20260923 20:13:31.240409 18306 fs_manager.cc:359] Metadata directory not provided
I20260923 20:13:31.240512 18306 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:13:31.246964 18306 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260923 20:13:31.250229 18321 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:13:31.251847 18306 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260923 20:13:31.252118 18306 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194121577027-1706-0/minicluster-data/master-0/data,/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194121577027-1706-0/minicluster-data/master-0/wal
uuid: "5cef025810a84435a7e1983ca1e0ac5b"
format_stamp: "Formatted at 2026-09-23 20:12:45 on dist-test-slave-th4g"
server_key: "1588b5bcbf8324cd5145e5581d5e4841"
server_key_iv: "c671879784aa95788080e2a19cc270a9"
server_key_version: "encryptionkey@0"
I20260923 20:13:31.253185 18306 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194121577027-1706-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194121577027-1706-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194121577027-1706-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:13:31.323760 18306 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:13:31.329531 18306 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:13:31.329916 18306 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:13:31.352084 18306 rpc_server.cc:307] RPC server started. Bound to: 127.1.170.190:44989
I20260923 20:13:31.352134 18372 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.170.190:44989 every 8 connection(s)
I20260923 20:13:31.353720 18306 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194121577027-1706-0/minicluster-data/master-0/data/info.pb
I20260923 20:13:31.362670 18373 sys_catalog.cc:263] Verifying existing consensus state
I20260923 20:13:31.363965 1706 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqUuku6/build/asan/bin/kudu as pid 18306
I20260923 20:13:31.365005 1706 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqUuku6/build/asan/bin/kudu
/tmp/dist-test-taskqUuku6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194121577027-1706-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194121577027-1706-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194121577027-1706-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194121577027-1706-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.170.129:43223
--local_ip_for_outbound_sockets=127.1.170.129
--tserver_master_addrs=127.1.170.190:44989
--webserver_port=36989
--webserver_interface=127.1.170.129
--keytab_file=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194121577027-1706-0/krb5kdc/kudu_127.1.170.129.keytab
--principal=kudu/127.1.170.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.170.148:42533
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194121577027-1706-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194121577027-1706-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194121577027-1706-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260923 20:13:31.367954 18373 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5cef025810a84435a7e1983ca1e0ac5b: Bootstrap starting.
I20260923 20:13:31.391599 18373 log.cc:824] T 00000000000000000000000000000000 P 5cef025810a84435a7e1983ca1e0ac5b: Log is configured to *not* fsync() on all Append() calls
I20260923 20:13:31.408823 18373 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5cef025810a84435a7e1983ca1e0ac5b: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260923 20:13:31.409422 18373 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5cef025810a84435a7e1983ca1e0ac5b: Bootstrap complete.
I20260923 20:13:31.419166 18373 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5cef025810a84435a7e1983ca1e0ac5b [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5cef025810a84435a7e1983ca1e0ac5b" member_type: VOTER last_known_addr { host: "127.1.170.190" port: 44989 } }
I20260923 20:13:31.420203 18373 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5cef025810a84435a7e1983ca1e0ac5b [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5cef025810a84435a7e1983ca1e0ac5b, State: Initialized, Role: FOLLOWER
I20260923 20:13:31.420967 18373 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5cef025810a84435a7e1983ca1e0ac5b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5cef025810a84435a7e1983ca1e0ac5b" member_type: VOTER last_known_addr { host: "127.1.170.190" port: 44989 } }
I20260923 20:13:31.421242 18373 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5cef025810a84435a7e1983ca1e0ac5b [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 20:13:31.421380 18373 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5cef025810a84435a7e1983ca1e0ac5b [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 20:13:31.421576 18373 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5cef025810a84435a7e1983ca1e0ac5b [term 2 FOLLOWER]: Advancing to term 3
I20260923 20:13:31.426354 18373 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5cef025810a84435a7e1983ca1e0ac5b [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5cef025810a84435a7e1983ca1e0ac5b" member_type: VOTER last_known_addr { host: "127.1.170.190" port: 44989 } }
I20260923 20:13:31.426895 18373 leader_election.cc:302] T 00000000000000000000000000000000 P 5cef025810a84435a7e1983ca1e0ac5b [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5cef025810a84435a7e1983ca1e0ac5b; no voters:
I20260923 20:13:31.427611 18373 leader_election.cc:288] T 00000000000000000000000000000000 P 5cef025810a84435a7e1983ca1e0ac5b [CANDIDATE]: Term 3 election: Requested vote from peers
I20260923 20:13:31.427759 18377 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5cef025810a84435a7e1983ca1e0ac5b [term 3 FOLLOWER]: Leader election won for term 3
I20260923 20:13:31.428854 18377 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5cef025810a84435a7e1983ca1e0ac5b [term 3 LEADER]: Becoming Leader. State: Replica: 5cef025810a84435a7e1983ca1e0ac5b, State: Running, Role: LEADER
I20260923 20:13:31.429525 18377 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5cef025810a84435a7e1983ca1e0ac5b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5cef025810a84435a7e1983ca1e0ac5b" member_type: VOTER last_known_addr { host: "127.1.170.190" port: 44989 } }
I20260923 20:13:31.430009 18373 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5cef025810a84435a7e1983ca1e0ac5b [sys.catalog]: configured and running, proceeding with master startup.
I20260923 20:13:31.432191 18379 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5cef025810a84435a7e1983ca1e0ac5b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5cef025810a84435a7e1983ca1e0ac5b. Latest consensus state: current_term: 3 leader_uuid: "5cef025810a84435a7e1983ca1e0ac5b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5cef025810a84435a7e1983ca1e0ac5b" member_type: VOTER last_known_addr { host: "127.1.170.190" port: 44989 } } }
I20260923 20:13:31.432586 18379 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5cef025810a84435a7e1983ca1e0ac5b [sys.catalog]: This master's current role is: LEADER
I20260923 20:13:31.432273 18378 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5cef025810a84435a7e1983ca1e0ac5b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "5cef025810a84435a7e1983ca1e0ac5b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5cef025810a84435a7e1983ca1e0ac5b" member_type: VOTER last_known_addr { host: "127.1.170.190" port: 44989 } } }
I20260923 20:13:31.432963 18378 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5cef025810a84435a7e1983ca1e0ac5b [sys.catalog]: This master's current role is: LEADER
I20260923 20:13:31.444198 18384 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 20:13:31.456069 18384 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 20:13:31.458379 18384 catalog_manager.cc:1259] Loaded cluster ID: e6eac8c738fb459ca0542b13a8d2eb00
I20260923 20:13:31.458567 18384 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 20:13:31.464310 18384 catalog_manager.cc:1504] Loading token signing keys...
I20260923 20:13:31.466998 18384 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 5cef025810a84435a7e1983ca1e0ac5b: Loaded TSK: 0
I20260923 20:13:31.467250 18384 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 5cef025810a84435a7e1983ca1e0ac5b: Loaded TSK: 1
F20260923 20:13:31.468801 18384 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f1f7c2729ad google::LogMessage::Fail() at ??:0
@ 0x7f1f7c27588a google::LogMessage::SendToLog() at ??:0
@ 0x7f1f7c2724de google::LogMessage::Flush() at ??:0
@ 0x7f1f7c273a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f1f7ffd67d2 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f1f7ffc573e kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7f1f91ca11c0 kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7f1f91ca0783 kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7f1f91d2c2a7 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_10::operator()() at ??:0
@ 0x7f1f91d2c0c4 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x7f1f91d765e9 std::function<>::operator()() at ??:0
@ 0x7f1f91ca4cdd kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7f1f91ca2c36 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f1f91d2a610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7f1f91d2a439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f1f7eebe345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f1f7eec191c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7f1f7eec1799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f1f7ee9ceaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7f1f821766db start_thread at ??:0
@ 0x7f1f7b10c61f clone at ??:0
W20260923 20:13:31.763409 18375 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:13:31.763818 18375 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:13:31.763988 18375 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:13:31.774281 18375 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:13:31.774438 18375 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:13:31.774637 18375 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.170.129
I20260923 20:13:31.784902 18375 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.170.148:42533
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194121577027-1706-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194121577027-1706-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194121577027-1706-0/krb5kdc/kudu_127.1.170.129.keytab
--principal=kudu/127.1.170.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.170.129:43223
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194121577027-1706-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.170.129
--webserver_port=36989
--tserver_master_addrs=127.1.170.190:44989
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.170.129
--log_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194121577027-1706-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:13:31.786381 18375 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:13:31.787701 18375 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:13:31.801465 18420 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:13:31.801429 18421 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:13:31.801563 18423 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:13:31.802191 18375 server_base.cc:1034] running on GCE node
I20260923 20:13:31.802785 18375 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:13:31.803689 18375 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:13:31.804942 18375 hybrid_clock.cc:648] HybridClock initialized: now 1790194411804872 us; error 68 us; skew 500 ppm
I20260923 20:13:31.805223 18375 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:13:31 dist-test-slave-th4g krb5kdc[17225](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194411, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.170.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:13:31.815006 18375 init.cc:377] Logged in from keytab as kudu/127.1.170.129@KRBTEST.COM (short username kudu)
I20260923 20:13:31.816704 18375 webserver.cc:460] Webserver started at http://127.1.170.129:36989/ using document root <none> and password file <none>
I20260923 20:13:31.817415 18375 fs_manager.cc:359] Metadata directory not provided
I20260923 20:13:31.817564 18375 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:13:31.822956 18375 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.001s
I20260923 20:13:31.826071 18430 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:13:31.827342 18375 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260923 20:13:31.827519 18375 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194121577027-1706-0/minicluster-data/ts-0/data,/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194121577027-1706-0/minicluster-data/ts-0/wal
uuid: "da1071f8ba4c4d93b8a8d1d6f143babe"
format_stamp: "Formatted at 2026-09-23 20:12:45 on dist-test-slave-th4g"
server_key: "e2497ed057070bf253ec73cee9ca10f9"
server_key_iv: "781814aef2a88d65a7440c984ab50e6c"
server_key_version: "encryptionkey@0"
I20260923 20:13:31.828483 18375 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194121577027-1706-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194121577027-1706-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194121577027-1706-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:13:31.850488 18375 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:13:31.855316 18375 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:13:31.855594 18375 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:13:31.856556 18375 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 20:13:31.858438 18437 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20260923 20:13:31.864099 18375 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260923 20:13:31.864238 18375 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.000s sys 0.000s
I20260923 20:13:31.864399 18375 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260923 20:13:31.867621 18375 ts_tablet_manager.cc:610] Registered 1 tablets
I20260923 20:13:31.867733 18375 ts_tablet_manager.cc:589] Time spent register tablets: real 0.003s user 0.002s sys 0.000s
I20260923 20:13:31.868109 18437 tablet_bootstrap.cc:492] T c6bf0a83f20343afa47ead1c6e7dfd8d P da1071f8ba4c4d93b8a8d1d6f143babe: Bootstrap starting.
I20260923 20:13:31.876585 18437 log.cc:824] T c6bf0a83f20343afa47ead1c6e7dfd8d P da1071f8ba4c4d93b8a8d1d6f143babe: Log is configured to *not* fsync() on all Append() calls
I20260923 20:13:31.892386 18437 tablet_bootstrap.cc:492] T c6bf0a83f20343afa47ead1c6e7dfd8d P da1071f8ba4c4d93b8a8d1d6f143babe: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260923 20:13:31.893323 18437 tablet_bootstrap.cc:492] T c6bf0a83f20343afa47ead1c6e7dfd8d P da1071f8ba4c4d93b8a8d1d6f143babe: Bootstrap complete.
I20260923 20:13:31.893965 18437 ts_tablet_manager.cc:1397] T c6bf0a83f20343afa47ead1c6e7dfd8d P da1071f8ba4c4d93b8a8d1d6f143babe: Time spent bootstrapping tablet: real 0.026s user 0.019s sys 0.005s
I20260923 20:13:31.902721 18437 raft_consensus.cc:348] T c6bf0a83f20343afa47ead1c6e7dfd8d P da1071f8ba4c4d93b8a8d1d6f143babe [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "908ce75423b74581b7328602946ef436" member_type: VOTER last_known_addr { host: "127.1.170.130" port: 35485 } } peers { permanent_uuid: "da1071f8ba4c4d93b8a8d1d6f143babe" member_type: VOTER last_known_addr { host: "127.1.170.129" port: 43223 } } peers { permanent_uuid: "b5dae6161ce245f8b7586278900d0d1e" member_type: VOTER last_known_addr { host: "127.1.170.131" port: 34241 } }
I20260923 20:13:31.904506 18437 raft_consensus.cc:729] T c6bf0a83f20343afa47ead1c6e7dfd8d P da1071f8ba4c4d93b8a8d1d6f143babe [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: da1071f8ba4c4d93b8a8d1d6f143babe, State: Initialized, Role: FOLLOWER
I20260923 20:13:31.905252 18437 consensus_queue.cc:260] T c6bf0a83f20343afa47ead1c6e7dfd8d P da1071f8ba4c4d93b8a8d1d6f143babe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "908ce75423b74581b7328602946ef436" member_type: VOTER last_known_addr { host: "127.1.170.130" port: 35485 } } peers { permanent_uuid: "da1071f8ba4c4d93b8a8d1d6f143babe" member_type: VOTER last_known_addr { host: "127.1.170.129" port: 43223 } } peers { permanent_uuid: "b5dae6161ce245f8b7586278900d0d1e" member_type: VOTER last_known_addr { host: "127.1.170.131" port: 34241 } }
I20260923 20:13:31.907032 18437 ts_tablet_manager.cc:1428] T c6bf0a83f20343afa47ead1c6e7dfd8d P da1071f8ba4c4d93b8a8d1d6f143babe: Time spent starting tablet: real 0.013s user 0.009s sys 0.003s
I20260923 20:13:31.915268 18375 rpc_server.cc:307] RPC server started. Bound to: 127.1.170.129:43223
I20260923 20:13:31.915303 18545 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.170.129:43223 every 8 connection(s)
I20260923 20:13:31.916596 18375 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194121577027-1706-0/minicluster-data/ts-0/data/info.pb
I20260923 20:13:31.921404 18548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:13:31.919483 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.190:44989 (local address 127.1.170.129:36197)
0923 20:13:31.920321 (+ 838us) negotiation.cc:107] Waiting for socket to connect
0923 20:13:31.920565 (+ 244us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.170.190:44989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":586,"thread_start_us":371,"threads_started":1}
W20260923 20:13:31.922148 18546 heartbeater.cc:644] Failed to heartbeat to 127.1.170.190:44989 (0 consecutive failures): Network error: Failed to ping master at 127.1.170.190:44989: Client connection negotiation failed: client connection to 127.1.170.190:44989: connect: Connection refused (error 111)
I20260923 20:13:31.923061 18548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:13:31.922689 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.190:44989 (local address 127.1.170.129:36771)
0923 20:13:31.922876 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0923 20:13:31.922947 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.170.190:44989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260923 20:13:31.923945 18548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:13:31.923589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.190:44989 (local address 127.1.170.129:48883)
0923 20:13:31.923737 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0923 20:13:31.923830 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.170.190:44989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260923 20:13:31.924199 18546 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260923 20:13:31.926734 1706 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqUuku6/build/asan/bin/kudu as pid 18375
I20260923 20:13:31.927364 1706 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqUuku6/build/asan/bin/kudu
/tmp/dist-test-taskqUuku6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194121577027-1706-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194121577027-1706-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194121577027-1706-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194121577027-1706-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.170.130:35485
--local_ip_for_outbound_sockets=127.1.170.130
--tserver_master_addrs=127.1.170.190:44989
--webserver_port=39021
--webserver_interface=127.1.170.130
--keytab_file=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194121577027-1706-0/krb5kdc/kudu_127.1.170.130.keytab
--principal=kudu/127.1.170.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.170.148:42533
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194121577027-1706-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194121577027-1706-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194121577027-1706-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:13:32.258591 18549 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:13:32.258881 18549 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:13:32.259055 18549 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:13:32.267745 18549 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:13:32.267853 18549 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:13:32.268059 18549 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.170.130
I20260923 20:13:32.278158 18549 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.170.148:42533
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194121577027-1706-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194121577027-1706-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194121577027-1706-0/krb5kdc/kudu_127.1.170.130.keytab
--principal=kudu/127.1.170.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.170.130:35485
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194121577027-1706-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.170.130
--webserver_port=39021
--tserver_master_addrs=127.1.170.190:44989
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.170.130
--log_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194121577027-1706-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:13:32.279420 18549 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:13:32.280673 18549 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:13:32.293802 18554 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:13:32.293893 18555 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:13:32.293852 18557 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:13:32.294637 18549 server_base.cc:1034] running on GCE node
I20260923 20:13:32.295269 18549 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:13:32.296237 18549 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:13:32.297490 18549 hybrid_clock.cc:648] HybridClock initialized: now 1790194412297439 us; error 43 us; skew 500 ppm
I20260923 20:13:32.297807 18549 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:13:32 dist-test-slave-th4g krb5kdc[17225](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194412, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.170.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:13:32.301002 18549 init.cc:377] Logged in from keytab as kudu/127.1.170.130@KRBTEST.COM (short username kudu)
I20260923 20:13:32.302553 18549 webserver.cc:460] Webserver started at http://127.1.170.130:39021/ using document root <none> and password file <none>
I20260923 20:13:32.303185 18549 fs_manager.cc:359] Metadata directory not provided
I20260923 20:13:32.303290 18549 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:13:32.308518 18549 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.003s
I20260923 20:13:32.311264 18564 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:13:32.312574 18549 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 20:13:32.312744 18549 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194121577027-1706-0/minicluster-data/ts-1/data,/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194121577027-1706-0/minicluster-data/ts-1/wal
uuid: "908ce75423b74581b7328602946ef436"
format_stamp: "Formatted at 2026-09-23 20:12:46 on dist-test-slave-th4g"
server_key: "f87b9f08f4d882504e9e47a613fa64a2"
server_key_iv: "dc05933e11d906b278f9186a65b2404a"
server_key_version: "encryptionkey@0"
I20260923 20:13:32.313686 18549 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194121577027-1706-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194121577027-1706-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194121577027-1706-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:13:32.328244 18549 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:13:32.332504 18549 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:13:32.332751 18549 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:13:32.333729 18549 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 20:13:32.335105 18549 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 20:13:32.335191 18549 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 20:13:32.335261 18549 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 20:13:32.335316 18549 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 20:13:32.369398 18549 rpc_server.cc:307] RPC server started. Bound to: 127.1.170.130:35485
I20260923 20:13:32.369503 18676 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.170.130:35485 every 8 connection(s)
I20260923 20:13:32.370935 18549 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194121577027-1706-0/minicluster-data/ts-1/data/info.pb
I20260923 20:13:32.371361 1706 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqUuku6/build/asan/bin/kudu as pid 18549
I20260923 20:13:32.372108 1706 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqUuku6/build/asan/bin/kudu
/tmp/dist-test-taskqUuku6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194121577027-1706-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194121577027-1706-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194121577027-1706-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194121577027-1706-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.170.131:34241
--local_ip_for_outbound_sockets=127.1.170.131
--tserver_master_addrs=127.1.170.190:44989
--webserver_port=34253
--webserver_interface=127.1.170.131
--keytab_file=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194121577027-1706-0/krb5kdc/kudu_127.1.170.131.keytab
--principal=kudu/127.1.170.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.170.148:42533
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194121577027-1706-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194121577027-1706-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194121577027-1706-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260923 20:13:32.376363 18679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:13:32.374431 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.190:44989 (local address 127.1.170.130:39751)
0923 20:13:32.375249 (+ 818us) negotiation.cc:107] Waiting for socket to connect
0923 20:13:32.375493 (+ 244us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.170.190:44989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":564,"thread_start_us":336,"threads_started":1}
W20260923 20:13:32.377075 18677 heartbeater.cc:644] Failed to heartbeat to 127.1.170.190:44989 (0 consecutive failures): Network error: Failed to ping master at 127.1.170.190:44989: Client connection negotiation failed: client connection to 127.1.170.190:44989: connect: Connection refused (error 111)
I20260923 20:13:32.378042 18679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:13:32.377663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.190:44989 (local address 127.1.170.130:46173)
0923 20:13:32.377808 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0923 20:13:32.377932 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.170.190:44989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260923 20:13:32.378948 18679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:13:32.378580 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.190:44989 (local address 127.1.170.130:52431)
0923 20:13:32.378745 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0923 20:13:32.378833 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.170.190:44989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260923 20:13:32.379271 18677 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260923 20:13:32.699332 18680 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:13:32.699604 18680 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:13:32.699754 18680 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:13:32.707214 18680 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:13:32.707319 18680 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:13:32.707518 18680 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.170.131
I20260923 20:13:32.717638 18680 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.170.148:42533
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194121577027-1706-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194121577027-1706-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194121577027-1706-0/krb5kdc/kudu_127.1.170.131.keytab
--principal=kudu/127.1.170.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.170.131:34241
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194121577027-1706-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.170.131
--webserver_port=34253
--tserver_master_addrs=127.1.170.190:44989
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.170.131
--log_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194121577027-1706-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:13:32.719046 18680 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:13:32.720314 18680 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:13:32.733268 18686 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:13:32.733319 18688 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:13:32.734025 18685 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:13:32.734351 18680 server_base.cc:1034] running on GCE node
I20260923 20:13:32.734969 18680 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:13:32.735900 18680 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:13:32.737143 18680 hybrid_clock.cc:648] HybridClock initialized: now 1790194412737090 us; error 47 us; skew 500 ppm
I20260923 20:13:32.737490 18680 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:13:32 dist-test-slave-th4g krb5kdc[17225](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194412, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.170.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:13:32.740725 18680 init.cc:377] Logged in from keytab as kudu/127.1.170.131@KRBTEST.COM (short username kudu)
I20260923 20:13:32.742297 18680 webserver.cc:460] Webserver started at http://127.1.170.131:34253/ using document root <none> and password file <none>
I20260923 20:13:32.743005 18680 fs_manager.cc:359] Metadata directory not provided
I20260923 20:13:32.743155 18680 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:13:32.748521 18680 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260923 20:13:32.751372 18695 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:13:32.752645 18680 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 20:13:32.752853 18680 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194121577027-1706-0/minicluster-data/ts-2/data,/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194121577027-1706-0/minicluster-data/ts-2/wal
uuid: "b5dae6161ce245f8b7586278900d0d1e"
format_stamp: "Formatted at 2026-09-23 20:12:46 on dist-test-slave-th4g"
server_key: "4501962af6ef3f402113111651fa2957"
server_key_iv: "d7e9d4f7ce561423488988002d01dd7e"
server_key_version: "encryptionkey@0"
I20260923 20:13:32.753906 18680 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194121577027-1706-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194121577027-1706-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194121577027-1706-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:13:32.773021 18680 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:13:32.777057 18680 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:13:32.777341 18680 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:13:32.778357 18680 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 20:13:32.779704 18680 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 20:13:32.779829 18680 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 20:13:32.779963 18680 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 20:13:32.780049 18680 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 20:13:32.815501 18680 rpc_server.cc:307] RPC server started. Bound to: 127.1.170.131:34241
I20260923 20:13:32.815550 18807 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.170.131:34241 every 8 connection(s)
I20260923 20:13:32.816958 18680 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194121577027-1706-0/minicluster-data/ts-2/data/info.pb
I20260923 20:13:32.818645 1706 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqUuku6/build/asan/bin/kudu as pid 18680
I20260923 20:13:32.822588 18811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:13:32.820455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.190:44989 (local address 127.1.170.131:54361)
0923 20:13:32.821262 (+ 807us) negotiation.cc:107] Waiting for socket to connect
0923 20:13:32.821489 (+ 227us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.170.190:44989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":570,"thread_start_us":329,"threads_started":1}
W20260923 20:13:32.823276 18808 heartbeater.cc:644] Failed to heartbeat to 127.1.170.190:44989 (0 consecutive failures): Network error: Failed to ping master at 127.1.170.190:44989: Client connection negotiation failed: client connection to 127.1.170.190:44989: connect: Connection refused (error 111)
I20260923 20:13:32.824175 18811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:13:32.823789 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.190:44989 (local address 127.1.170.131:47205)
0923 20:13:32.823989 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0923 20:13:32.824060 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.170.190:44989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98}
I20260923 20:13:32.825066 18811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:13:32.824743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.190:44989 (local address 127.1.170.131:60647)
0923 20:13:32.824880 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0923 20:13:32.824956 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.170.190:44989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260923 20:13:32.825340 18808 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260923 20:13:32.925520 18812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:13:32.924727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.190:44989 (local address 127.1.170.129:53527)
0923 20:13:32.925195 (+ 468us) negotiation.cc:107] Waiting for socket to connect
0923 20:13:32.925273 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.170.190:44989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":362,"thread_start_us":234,"threads_started":1}
I20260923 20:13:33.256538 18813 raft_consensus.cc:482] T c6bf0a83f20343afa47ead1c6e7dfd8d P da1071f8ba4c4d93b8a8d1d6f143babe [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 20:13:33.256875 18813 raft_consensus.cc:504] T c6bf0a83f20343afa47ead1c6e7dfd8d P da1071f8ba4c4d93b8a8d1d6f143babe [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "908ce75423b74581b7328602946ef436" member_type: VOTER last_known_addr { host: "127.1.170.130" port: 35485 } } peers { permanent_uuid: "da1071f8ba4c4d93b8a8d1d6f143babe" member_type: VOTER last_known_addr { host: "127.1.170.129" port: 43223 } } peers { permanent_uuid: "b5dae6161ce245f8b7586278900d0d1e" member_type: VOTER last_known_addr { host: "127.1.170.131" port: 34241 } }
I20260923 20:13:33.258656 18813 leader_election.cc:288] T c6bf0a83f20343afa47ead1c6e7dfd8d P da1071f8ba4c4d93b8a8d1d6f143babe [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 908ce75423b74581b7328602946ef436 (127.1.170.130:35485), b5dae6161ce245f8b7586278900d0d1e (127.1.170.131:34241)
Sep 23 20:13:33 dist-test-slave-th4g krb5kdc[17225](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194411, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.170.129@KRBTEST.COM for kudu/127.1.170.131@KRBTEST.COM
Sep 23 20:13:33 dist-test-slave-th4g krb5kdc[17225](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194411, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.170.129@KRBTEST.COM for kudu/127.1.170.130@KRBTEST.COM
I20260923 20:13:33.282163 18812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:13:33.258905 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.131:34241 (local address 127.1.170.129:35907)
0923 20:13:33.259155 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0923 20:13:33.259195 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0923 20:13:33.259780 (+ 585us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:13:33.263754 (+ 3974us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:13:33.263763 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:13:33.264548 (+ 785us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:13:33.265288 (+ 740us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:13:33.265307 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:13:33.266898 (+ 1591us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:13:33.266902 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:13:33.269507 (+ 2605us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:13:33.269523 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:13:33.270643 (+ 1120us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:13:33.270649 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:13:33.270919 (+ 270us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:13:33.271760 (+ 841us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:13:33.271790 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:13:33.274444 (+ 2654us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:13:33.278063 (+ 3619us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:13:33.278073 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:13:33.278089 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:13:33.278727 (+ 638us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:13:33.279210 (+ 483us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:13:33.279217 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:13:33.279221 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:13:33.279432 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:13:33.280398 (+ 966us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:13:33.280408 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:13:33.280757 (+ 349us) client_negotiation.cc:769] Sending connection context
0923 20:13:33.281084 (+ 327us) client_negotiation.cc:240] Negotiation successful
0923 20:13:33.281713 (+ 629us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"mutex_wait_us":338,"spinlock_wait_cycles":4992}
I20260923 20:13:33.282202 18816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:13:33.259366 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.170.129:35907 (local address 127.1.170.131:34241)
0923 20:13:33.261093 (+ 1727us) server_negotiation.cc:206] Beginning negotiation
0923 20:13:33.261101 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 20:13:33.261134 (+ 33us) server_negotiation.cc:407] Connection header received
0923 20:13:33.262791 (+ 1657us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:13:33.262799 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:13:33.263261 (+ 462us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:13:33.263629 (+ 368us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:13:33.265482 (+ 1853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:13:33.266689 (+ 1207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:13:33.269683 (+ 2994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:13:33.270422 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:13:33.270639 (+ 217us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:13:33.274758 (+ 4119us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:13:33.274767 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:13:33.274781 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:13:33.274825 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:13:33.277817 (+ 2992us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:13:33.278862 (+ 1045us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:13:33.278869 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:13:33.278874 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:13:33.279078 (+ 204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:13:33.279567 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:13:33.279572 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:13:33.279576 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:13:33.280196 (+ 620us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:13:33.280334 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0923 20:13:33.281368 (+ 1034us) server_negotiation.cc:299] Negotiation successful
0923 20:13:33.281750 (+ 382us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1587,"thread_start_us":1483,"threads_started":1}
I20260923 20:13:33.282733 18814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:13:33.258943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.130:35485 (local address 127.1.170.129:53731)
0923 20:13:33.260941 (+ 1998us) negotiation.cc:107] Waiting for socket to connect
0923 20:13:33.260956 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0923 20:13:33.261113 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:13:33.263754 (+ 2641us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:13:33.263765 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:13:33.264549 (+ 784us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:13:33.265287 (+ 738us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:13:33.265307 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:13:33.266892 (+ 1585us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:13:33.266897 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:13:33.269505 (+ 2608us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:13:33.269522 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:13:33.271112 (+ 1590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:13:33.271116 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:13:33.271314 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:13:33.271923 (+ 609us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:13:33.271941 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:13:33.275169 (+ 3228us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:13:33.279218 (+ 4049us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:13:33.279222 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:13:33.279224 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:13:33.279736 (+ 512us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:13:33.280432 (+ 696us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:13:33.280436 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:13:33.280438 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:13:33.280586 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:13:33.281679 (+ 1093us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:13:33.281685 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:13:33.281952 (+ 267us) client_negotiation.cc:769] Sending connection context
0923 20:13:33.282151 (+ 199us) client_negotiation.cc:240] Negotiation successful
0923 20:13:33.282417 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1859,"mutex_wait_us":245,"spinlock_wait_cycles":45568,"thread_start_us":1699,"threads_started":1}
I20260923 20:13:33.283325 18815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:13:33.259223 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.170.129:53731 (local address 127.1.170.130:35485)
0923 20:13:33.260666 (+ 1443us) server_negotiation.cc:206] Beginning negotiation
0923 20:13:33.260685 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0923 20:13:33.261074 (+ 389us) server_negotiation.cc:407] Connection header received
0923 20:13:33.262777 (+ 1703us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:13:33.262788 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:13:33.263268 (+ 480us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:13:33.263650 (+ 382us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:13:33.265483 (+ 1833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:13:33.266700 (+ 1217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:13:33.269691 (+ 2991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:13:33.270716 (+ 1025us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:13:33.270828 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:13:33.275480 (+ 4652us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:13:33.275494 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:13:33.275510 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:13:33.275569 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:13:33.279082 (+ 3513us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:13:33.279874 (+ 792us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:13:33.279880 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:13:33.279884 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:13:33.280045 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:13:33.280704 (+ 659us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:13:33.280709 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:13:33.280712 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:13:33.281479 (+ 767us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:13:33.281696 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0923 20:13:33.282476 (+ 780us) server_negotiation.cc:299] Negotiation successful
0923 20:13:33.282875 (+ 399us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1207,"thread_start_us":1034,"threads_started":1}
W20260923 20:13:33.286177 18431 leader_election.cc:341] T c6bf0a83f20343afa47ead1c6e7dfd8d P da1071f8ba4c4d93b8a8d1d6f143babe [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b5dae6161ce245f8b7586278900d0d1e (127.1.170.131:34241): Not found: Tablet not found: c6bf0a83f20343afa47ead1c6e7dfd8d
W20260923 20:13:33.286468 18434 leader_election.cc:341] T c6bf0a83f20343afa47ead1c6e7dfd8d P da1071f8ba4c4d93b8a8d1d6f143babe [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 908ce75423b74581b7328602946ef436 (127.1.170.130:35485): Not found: Tablet not found: c6bf0a83f20343afa47ead1c6e7dfd8d
I20260923 20:13:33.286597 18434 leader_election.cc:302] T c6bf0a83f20343afa47ead1c6e7dfd8d P da1071f8ba4c4d93b8a8d1d6f143babe [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: da1071f8ba4c4d93b8a8d1d6f143babe; no voters: 908ce75423b74581b7328602946ef436, b5dae6161ce245f8b7586278900d0d1e
I20260923 20:13:33.287060 18813 raft_consensus.cc:2726] T c6bf0a83f20343afa47ead1c6e7dfd8d P da1071f8ba4c4d93b8a8d1d6f143babe [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260923 20:13:33.380685 18817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:13:33.379779 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.190:44989 (local address 127.1.170.130:49997)
0923 20:13:33.380266 (+ 487us) negotiation.cc:107] Waiting for socket to connect
0923 20:13:33.380446 (+ 180us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.170.190:44989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":381,"thread_start_us":235,"threads_started":1}
I20260923 20:13:33.826822 18818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:13:33.825959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.190:44989 (local address 127.1.170.131:43809)
0923 20:13:33.826488 (+ 529us) negotiation.cc:107] Waiting for socket to connect
0923 20:13:33.826597 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.170.190:44989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":416,"thread_start_us":282,"threads_started":1}
I20260923 20:13:33.927379 18819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:13:33.926584 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.190:44989 (local address 127.1.170.129:56067)
0923 20:13:33.927119 (+ 535us) negotiation.cc:107] Waiting for socket to connect
0923 20:13:33.927219 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.170.190:44989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":415,"thread_start_us":300,"threads_started":1}
I20260923 20:13:34.382547 18820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:13:34.381700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.190:44989 (local address 127.1.170.130:57885)
0923 20:13:34.382277 (+ 577us) negotiation.cc:107] Waiting for socket to connect
0923 20:13:34.382374 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.170.190:44989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":457,"thread_start_us":345,"threads_started":1}
I20260923 20:13:34.828630 18821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:13:34.827848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.190:44989 (local address 127.1.170.131:33713)
0923 20:13:34.828349 (+ 501us) negotiation.cc:107] Waiting for socket to connect
0923 20:13:34.828448 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.170.190:44989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":372,"thread_start_us":253,"threads_started":1}
I20260923 20:13:34.929518 18822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:13:34.928420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.190:44989 (local address 127.1.170.129:54635)
0923 20:13:34.929149 (+ 729us) negotiation.cc:107] Waiting for socket to connect
0923 20:13:34.929273 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.170.190:44989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":560,"thread_start_us":359,"threads_started":1}
I20260923 20:13:35.018237 18823 raft_consensus.cc:482] T c6bf0a83f20343afa47ead1c6e7dfd8d P da1071f8ba4c4d93b8a8d1d6f143babe [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 20:13:35.018563 18823 raft_consensus.cc:504] T c6bf0a83f20343afa47ead1c6e7dfd8d P da1071f8ba4c4d93b8a8d1d6f143babe [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "908ce75423b74581b7328602946ef436" member_type: VOTER last_known_addr { host: "127.1.170.130" port: 35485 } } peers { permanent_uuid: "da1071f8ba4c4d93b8a8d1d6f143babe" member_type: VOTER last_known_addr { host: "127.1.170.129" port: 43223 } } peers { permanent_uuid: "b5dae6161ce245f8b7586278900d0d1e" member_type: VOTER last_known_addr { host: "127.1.170.131" port: 34241 } }
I20260923 20:13:35.019253 18823 leader_election.cc:288] T c6bf0a83f20343afa47ead1c6e7dfd8d P da1071f8ba4c4d93b8a8d1d6f143babe [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 908ce75423b74581b7328602946ef436 (127.1.170.130:35485), b5dae6161ce245f8b7586278900d0d1e (127.1.170.131:34241)
W20260923 20:13:35.020532 18431 leader_election.cc:341] T c6bf0a83f20343afa47ead1c6e7dfd8d P da1071f8ba4c4d93b8a8d1d6f143babe [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b5dae6161ce245f8b7586278900d0d1e (127.1.170.131:34241): Not found: Tablet not found: c6bf0a83f20343afa47ead1c6e7dfd8d
W20260923 20:13:35.020745 18434 leader_election.cc:341] T c6bf0a83f20343afa47ead1c6e7dfd8d P da1071f8ba4c4d93b8a8d1d6f143babe [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 908ce75423b74581b7328602946ef436 (127.1.170.130:35485): Not found: Tablet not found: c6bf0a83f20343afa47ead1c6e7dfd8d
I20260923 20:13:35.020843 18434 leader_election.cc:302] T c6bf0a83f20343afa47ead1c6e7dfd8d P da1071f8ba4c4d93b8a8d1d6f143babe [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: da1071f8ba4c4d93b8a8d1d6f143babe; no voters: 908ce75423b74581b7328602946ef436, b5dae6161ce245f8b7586278900d0d1e
I20260923 20:13:35.021113 18823 raft_consensus.cc:2726] T c6bf0a83f20343afa47ead1c6e7dfd8d P da1071f8ba4c4d93b8a8d1d6f143babe [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260923 20:13:35.385188 18824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:13:35.384052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.190:44989 (local address 127.1.170.130:55601)
0923 20:13:35.384898 (+ 846us) negotiation.cc:107] Waiting for socket to connect
0923 20:13:35.385013 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.170.190:44989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":588,"thread_start_us":467,"threads_started":1}
I20260923 20:13:35.831025 18825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:13:35.829722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.190:44989 (local address 127.1.170.131:56851)
0923 20:13:35.830502 (+ 780us) negotiation.cc:107] Waiting for socket to connect
0923 20:13:35.830840 (+ 338us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.170.190:44989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":627,"thread_start_us":443,"threads_started":1}
I20260923 20:13:35.931713 18826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:13:35.930935 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.190:44989 (local address 127.1.170.129:37149)
0923 20:13:35.931421 (+ 486us) negotiation.cc:107] Waiting for socket to connect
0923 20:13:35.931512 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.170.190:44989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":371,"thread_start_us":273,"threads_started":1}
I20260923 20:13:36.387115 18827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:13:36.386339 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.190:44989 (local address 127.1.170.130:35371)
0923 20:13:36.386840 (+ 501us) negotiation.cc:107] Waiting for socket to connect
0923 20:13:36.386930 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.170.190:44989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":384,"thread_start_us":253,"threads_started":1}
I20260923 20:13:36.833088 18828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:13:36.832127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.190:44989 (local address 127.1.170.131:52127)
0923 20:13:36.832650 (+ 523us) negotiation.cc:107] Waiting for socket to connect
0923 20:13:36.832737 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.170.190:44989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":403,"thread_start_us":267,"threads_started":1}
I20260923 20:13:36.933643 18829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:13:36.932785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.190:44989 (local address 127.1.170.129:57013)
0923 20:13:36.933367 (+ 582us) negotiation.cc:107] Waiting for socket to connect
0923 20:13:36.933464 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.170.190:44989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":424,"thread_start_us":286,"threads_started":1}
I20260923 20:13:37.388988 18830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:13:37.388130 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.190:44989 (local address 127.1.170.130:36753)
0923 20:13:37.388688 (+ 558us) negotiation.cc:107] Waiting for socket to connect
0923 20:13:37.388774 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.170.190:44989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":435,"thread_start_us":308,"threads_started":1}
I20260923 20:13:37.807690 18831 raft_consensus.cc:482] T c6bf0a83f20343afa47ead1c6e7dfd8d P da1071f8ba4c4d93b8a8d1d6f143babe [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 20:13:37.807868 18831 raft_consensus.cc:504] T c6bf0a83f20343afa47ead1c6e7dfd8d P da1071f8ba4c4d93b8a8d1d6f143babe [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "908ce75423b74581b7328602946ef436" member_type: VOTER last_known_addr { host: "127.1.170.130" port: 35485 } } peers { permanent_uuid: "da1071f8ba4c4d93b8a8d1d6f143babe" member_type: VOTER last_known_addr { host: "127.1.170.129" port: 43223 } } peers { permanent_uuid: "b5dae6161ce245f8b7586278900d0d1e" member_type: VOTER last_known_addr { host: "127.1.170.131" port: 34241 } }
I20260923 20:13:37.808458 18831 leader_election.cc:288] T c6bf0a83f20343afa47ead1c6e7dfd8d P da1071f8ba4c4d93b8a8d1d6f143babe [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 908ce75423b74581b7328602946ef436 (127.1.170.130:35485), b5dae6161ce245f8b7586278900d0d1e (127.1.170.131:34241)
W20260923 20:13:37.809478 18431 leader_election.cc:341] T c6bf0a83f20343afa47ead1c6e7dfd8d P da1071f8ba4c4d93b8a8d1d6f143babe [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b5dae6161ce245f8b7586278900d0d1e (127.1.170.131:34241): Not found: Tablet not found: c6bf0a83f20343afa47ead1c6e7dfd8d
W20260923 20:13:37.809657 18434 leader_election.cc:341] T c6bf0a83f20343afa47ead1c6e7dfd8d P da1071f8ba4c4d93b8a8d1d6f143babe [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 908ce75423b74581b7328602946ef436 (127.1.170.130:35485): Not found: Tablet not found: c6bf0a83f20343afa47ead1c6e7dfd8d
I20260923 20:13:37.809810 18434 leader_election.cc:302] T c6bf0a83f20343afa47ead1c6e7dfd8d P da1071f8ba4c4d93b8a8d1d6f143babe [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: da1071f8ba4c4d93b8a8d1d6f143babe; no voters: 908ce75423b74581b7328602946ef436, b5dae6161ce245f8b7586278900d0d1e
I20260923 20:13:37.810134 18831 raft_consensus.cc:2726] T c6bf0a83f20343afa47ead1c6e7dfd8d P da1071f8ba4c4d93b8a8d1d6f143babe [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260923 20:13:37.834996 18832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:13:37.834250 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.190:44989 (local address 127.1.170.131:58691)
0923 20:13:37.834696 (+ 446us) negotiation.cc:107] Waiting for socket to connect
0923 20:13:37.834789 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.170.190:44989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":336,"thread_start_us":228,"threads_started":1}
I20260923 20:13:37.935513 18833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:13:37.934681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.190:44989 (local address 127.1.170.129:42995)
0923 20:13:37.935207 (+ 526us) negotiation.cc:107] Waiting for socket to connect
0923 20:13:37.935313 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.170.190:44989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":418,"thread_start_us":287,"threads_started":1}
I20260923 20:13:38.390951 18834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:13:38.390123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.190:44989 (local address 127.1.170.130:40535)
0923 20:13:38.390669 (+ 546us) negotiation.cc:107] Waiting for socket to connect
0923 20:13:38.390758 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.170.190:44989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":435,"thread_start_us":286,"threads_started":1}
I20260923 20:13:38.837028 18835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:13:38.836228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.190:44989 (local address 127.1.170.131:50859)
0923 20:13:38.836756 (+ 528us) negotiation.cc:107] Waiting for socket to connect
0923 20:13:38.836846 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.170.190:44989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":414,"thread_start_us":291,"threads_started":1}
I20260923 20:13:38.938230 18836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:13:38.936547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.190:44989 (local address 127.1.170.129:49531)
0923 20:13:38.937532 (+ 985us) negotiation.cc:107] Waiting for socket to connect
0923 20:13:38.937681 (+ 149us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.170.190:44989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":792,"thread_start_us":394,"threads_started":1}
I20260923 20:13:39.394377 18837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:13:39.392924 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.190:44989 (local address 127.1.170.130:59825)
0923 20:13:39.393787 (+ 863us) negotiation.cc:107] Waiting for socket to connect
0923 20:13:39.394033 (+ 246us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.170.190:44989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":693,"thread_start_us":423,"threads_started":1}
I20260923 20:13:39.768903 18838 raft_consensus.cc:482] T c6bf0a83f20343afa47ead1c6e7dfd8d P da1071f8ba4c4d93b8a8d1d6f143babe [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 20:13:39.769193 18838 raft_consensus.cc:504] T c6bf0a83f20343afa47ead1c6e7dfd8d P da1071f8ba4c4d93b8a8d1d6f143babe [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "908ce75423b74581b7328602946ef436" member_type: VOTER last_known_addr { host: "127.1.170.130" port: 35485 } } peers { permanent_uuid: "da1071f8ba4c4d93b8a8d1d6f143babe" member_type: VOTER last_known_addr { host: "127.1.170.129" port: 43223 } } peers { permanent_uuid: "b5dae6161ce245f8b7586278900d0d1e" member_type: VOTER last_known_addr { host: "127.1.170.131" port: 34241 } }
I20260923 20:13:39.769843 18838 leader_election.cc:288] T c6bf0a83f20343afa47ead1c6e7dfd8d P da1071f8ba4c4d93b8a8d1d6f143babe [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 908ce75423b74581b7328602946ef436 (127.1.170.130:35485), b5dae6161ce245f8b7586278900d0d1e (127.1.170.131:34241)
W20260923 20:13:39.770825 18434 leader_election.cc:341] T c6bf0a83f20343afa47ead1c6e7dfd8d P da1071f8ba4c4d93b8a8d1d6f143babe [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 908ce75423b74581b7328602946ef436 (127.1.170.130:35485): Not found: Tablet not found: c6bf0a83f20343afa47ead1c6e7dfd8d
W20260923 20:13:39.771004 18431 leader_election.cc:341] T c6bf0a83f20343afa47ead1c6e7dfd8d P da1071f8ba4c4d93b8a8d1d6f143babe [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b5dae6161ce245f8b7586278900d0d1e (127.1.170.131:34241): Not found: Tablet not found: c6bf0a83f20343afa47ead1c6e7dfd8d
I20260923 20:13:39.771143 18431 leader_election.cc:302] T c6bf0a83f20343afa47ead1c6e7dfd8d P da1071f8ba4c4d93b8a8d1d6f143babe [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: da1071f8ba4c4d93b8a8d1d6f143babe; no voters: 908ce75423b74581b7328602946ef436, b5dae6161ce245f8b7586278900d0d1e
I20260923 20:13:39.771402 18838 raft_consensus.cc:2726] T c6bf0a83f20343afa47ead1c6e7dfd8d P da1071f8ba4c4d93b8a8d1d6f143babe [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260923 20:13:39.839102 18839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:13:39.838198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.190:44989 (local address 127.1.170.131:41867)
0923 20:13:39.838783 (+ 585us) negotiation.cc:107] Waiting for socket to connect
0923 20:13:39.838874 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.170.190:44989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":469,"thread_start_us":277,"threads_started":1}
I20260923 20:13:39.940529 18840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:13:39.939448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.190:44989 (local address 127.1.170.129:55585)
0923 20:13:39.940096 (+ 648us) negotiation.cc:107] Waiting for socket to connect
0923 20:13:39.940324 (+ 228us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.170.190:44989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":533,"thread_start_us":379,"threads_started":1}
I20260923 20:13:40.396435 18841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:13:40.395622 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.190:44989 (local address 127.1.170.130:52957)
0923 20:13:40.396136 (+ 514us) negotiation.cc:107] Waiting for socket to connect
0923 20:13:40.396231 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.170.190:44989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":395,"thread_start_us":269,"threads_started":1}
I20260923 20:13:40.840973 18842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:13:40.840197 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.190:44989 (local address 127.1.170.131:40199)
0923 20:13:40.840696 (+ 499us) negotiation.cc:107] Waiting for socket to connect
0923 20:13:40.840788 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.170.190:44989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":395,"thread_start_us":275,"threads_started":1}
I20260923 20:13:40.942432 18843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:13:40.941543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.190:44989 (local address 127.1.170.129:43007)
0923 20:13:40.942147 (+ 604us) negotiation.cc:107] Waiting for socket to connect
0923 20:13:40.942236 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.170.190:44989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":506,"thread_start_us":267,"threads_started":1}
I20260923 20:13:41.398571 18844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:13:41.397709 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.190:44989 (local address 127.1.170.130:45421)
0923 20:13:41.398313 (+ 604us) negotiation.cc:107] Waiting for socket to connect
0923 20:13:41.398403 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.170.190:44989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":489,"thread_start_us":344,"threads_started":1}
I20260923 20:13:41.842725 18845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:13:41.842022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.190:44989 (local address 127.1.170.131:53905)
0923 20:13:41.842492 (+ 470us) negotiation.cc:107] Waiting for socket to connect
0923 20:13:41.842573 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.170.190:44989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":359,"thread_start_us":225,"threads_started":1}
I20260923 20:13:41.944221 18846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:13:41.943515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.190:44989 (local address 127.1.170.129:33561)
0923 20:13:41.943942 (+ 427us) negotiation.cc:107] Waiting for socket to connect
0923 20:13:41.944061 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.170.190:44989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":338,"thread_start_us":240,"threads_started":1}
I20260923 20:13:42.176205 18847 raft_consensus.cc:482] T c6bf0a83f20343afa47ead1c6e7dfd8d P da1071f8ba4c4d93b8a8d1d6f143babe [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 20:13:42.176476 18847 raft_consensus.cc:504] T c6bf0a83f20343afa47ead1c6e7dfd8d P da1071f8ba4c4d93b8a8d1d6f143babe [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "908ce75423b74581b7328602946ef436" member_type: VOTER last_known_addr { host: "127.1.170.130" port: 35485 } } peers { permanent_uuid: "da1071f8ba4c4d93b8a8d1d6f143babe" member_type: VOTER last_known_addr { host: "127.1.170.129" port: 43223 } } peers { permanent_uuid: "b5dae6161ce245f8b7586278900d0d1e" member_type: VOTER last_known_addr { host: "127.1.170.131" port: 34241 } }
I20260923 20:13:42.177138 18847 leader_election.cc:288] T c6bf0a83f20343afa47ead1c6e7dfd8d P da1071f8ba4c4d93b8a8d1d6f143babe [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 908ce75423b74581b7328602946ef436 (127.1.170.130:35485), b5dae6161ce245f8b7586278900d0d1e (127.1.170.131:34241)
W20260923 20:13:42.178179 18431 leader_election.cc:341] T c6bf0a83f20343afa47ead1c6e7dfd8d P da1071f8ba4c4d93b8a8d1d6f143babe [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b5dae6161ce245f8b7586278900d0d1e (127.1.170.131:34241): Not found: Tablet not found: c6bf0a83f20343afa47ead1c6e7dfd8d
W20260923 20:13:42.178397 18434 leader_election.cc:341] T c6bf0a83f20343afa47ead1c6e7dfd8d P da1071f8ba4c4d93b8a8d1d6f143babe [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 908ce75423b74581b7328602946ef436 (127.1.170.130:35485): Not found: Tablet not found: c6bf0a83f20343afa47ead1c6e7dfd8d
I20260923 20:13:42.178508 18434 leader_election.cc:302] T c6bf0a83f20343afa47ead1c6e7dfd8d P da1071f8ba4c4d93b8a8d1d6f143babe [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: da1071f8ba4c4d93b8a8d1d6f143babe; no voters: 908ce75423b74581b7328602946ef436, b5dae6161ce245f8b7586278900d0d1e
I20260923 20:13:42.178814 18847 raft_consensus.cc:2726] T c6bf0a83f20343afa47ead1c6e7dfd8d P da1071f8ba4c4d93b8a8d1d6f143babe [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260923 20:13:42.400590 18848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:13:42.399644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.190:44989 (local address 127.1.170.130:60617)
0923 20:13:42.400253 (+ 609us) negotiation.cc:107] Waiting for socket to connect
0923 20:13:42.400385 (+ 132us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.170.190:44989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":413,"thread_start_us":299,"threads_started":1}
I20260923 20:13:42.844702 18849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:13:42.843967 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.190:44989 (local address 127.1.170.131:47075)
0923 20:13:42.844433 (+ 466us) negotiation.cc:107] Waiting for socket to connect
0923 20:13:42.844523 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.170.190:44989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":361,"thread_start_us":256,"threads_started":1}
I20260923 20:13:42.945930 18850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:13:42.945210 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.190:44989 (local address 127.1.170.129:42239)
0923 20:13:42.945646 (+ 436us) negotiation.cc:107] Waiting for socket to connect
0923 20:13:42.945727 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.170.190:44989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":341,"thread_start_us":234,"threads_started":1}
I20260923 20:13:43.402459 18851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:13:43.401660 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.190:44989 (local address 127.1.170.130:49965)
0923 20:13:43.402193 (+ 533us) negotiation.cc:107] Waiting for socket to connect
0923 20:13:43.402279 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.170.190:44989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":423,"thread_start_us":282,"threads_started":1}
I20260923 20:13:43.846650 18852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:13:43.845813 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.190:44989 (local address 127.1.170.131:52257)
0923 20:13:43.846408 (+ 595us) negotiation.cc:107] Waiting for socket to connect
0923 20:13:43.846488 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.170.190:44989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":484,"thread_start_us":324,"threads_started":1}
I20260923 20:13:43.948242 18853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:13:43.947210 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.190:44989 (local address 127.1.170.129:40369)
0923 20:13:43.947883 (+ 673us) negotiation.cc:107] Waiting for socket to connect
0923 20:13:43.948017 (+ 134us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.170.190:44989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":516,"thread_start_us":363,"threads_started":1}
I20260923 20:13:44.404376 18854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:13:44.403547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.190:44989 (local address 127.1.170.130:49697)
0923 20:13:44.404078 (+ 531us) negotiation.cc:107] Waiting for socket to connect
0923 20:13:44.404169 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.170.190:44989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":388,"thread_start_us":264,"threads_started":1}
I20260923 20:13:44.848394 18855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:13:44.847638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.190:44989 (local address 127.1.170.131:57201)
0923 20:13:44.848115 (+ 477us) negotiation.cc:107] Waiting for socket to connect
0923 20:13:44.848200 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.170.190:44989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":378,"thread_start_us":256,"threads_started":1}
I20260923 20:13:44.950553 18856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:13:44.949687 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.190:44989 (local address 127.1.170.129:38619)
0923 20:13:44.950256 (+ 569us) negotiation.cc:107] Waiting for socket to connect
0923 20:13:44.950361 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.170.190:44989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":460,"thread_start_us":338,"threads_started":1}
I20260923 20:13:45.406195 18857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:13:45.405381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.190:44989 (local address 127.1.170.130:43903)
0923 20:13:45.405924 (+ 543us) negotiation.cc:107] Waiting for socket to connect
0923 20:13:45.406019 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.170.190:44989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":378,"thread_start_us":262,"threads_started":1}
I20260923 20:13:45.850401 18858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:13:45.849506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.190:44989 (local address 127.1.170.131:48407)
0923 20:13:45.850132 (+ 626us) negotiation.cc:107] Waiting for socket to connect
0923 20:13:45.850238 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.170.190:44989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":513,"thread_start_us":278,"threads_started":1}
I20260923 20:13:45.952316 18859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:13:45.951579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.190:44989 (local address 127.1.170.129:45785)
0923 20:13:45.952044 (+ 465us) negotiation.cc:107] Waiting for socket to connect
0923 20:13:45.952135 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.170.190:44989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":363,"thread_start_us":235,"threads_started":1}
I20260923 20:13:46.407990 18860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:13:46.407207 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.190:44989 (local address 127.1.170.130:49133)
0923 20:13:46.407706 (+ 499us) negotiation.cc:107] Waiting for socket to connect
0923 20:13:46.407787 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.170.190:44989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":401,"thread_start_us":276,"threads_started":1}
I20260923 20:13:46.852268 18861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:13:46.851460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.190:44989 (local address 127.1.170.131:37179)
0923 20:13:46.851982 (+ 522us) negotiation.cc:107] Waiting for socket to connect
0923 20:13:46.852070 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.170.190:44989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":375,"thread_start_us":236,"threads_started":1}
I20260923 20:13:46.954572 18862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:13:46.953706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.190:44989 (local address 127.1.170.129:49355)
0923 20:13:46.954278 (+ 572us) negotiation.cc:107] Waiting for socket to connect
0923 20:13:46.954358 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.170.190:44989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":473,"thread_start_us":333,"threads_started":1}
I20260923 20:13:47.013571 18863 raft_consensus.cc:482] T c6bf0a83f20343afa47ead1c6e7dfd8d P da1071f8ba4c4d93b8a8d1d6f143babe [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 20:13:47.013834 18863 raft_consensus.cc:504] T c6bf0a83f20343afa47ead1c6e7dfd8d P da1071f8ba4c4d93b8a8d1d6f143babe [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "908ce75423b74581b7328602946ef436" member_type: VOTER last_known_addr { host: "127.1.170.130" port: 35485 } } peers { permanent_uuid: "da1071f8ba4c4d93b8a8d1d6f143babe" member_type: VOTER last_known_addr { host: "127.1.170.129" port: 43223 } } peers { permanent_uuid: "b5dae6161ce245f8b7586278900d0d1e" member_type: VOTER last_known_addr { host: "127.1.170.131" port: 34241 } }
I20260923 20:13:47.014905 18863 leader_election.cc:288] T c6bf0a83f20343afa47ead1c6e7dfd8d P da1071f8ba4c4d93b8a8d1d6f143babe [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 908ce75423b74581b7328602946ef436 (127.1.170.130:35485), b5dae6161ce245f8b7586278900d0d1e (127.1.170.131:34241)
W20260923 20:13:47.015869 18431 leader_election.cc:341] T c6bf0a83f20343afa47ead1c6e7dfd8d P da1071f8ba4c4d93b8a8d1d6f143babe [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b5dae6161ce245f8b7586278900d0d1e (127.1.170.131:34241): Not found: Tablet not found: c6bf0a83f20343afa47ead1c6e7dfd8d
W20260923 20:13:47.016098 18434 leader_election.cc:341] T c6bf0a83f20343afa47ead1c6e7dfd8d P da1071f8ba4c4d93b8a8d1d6f143babe [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 908ce75423b74581b7328602946ef436 (127.1.170.130:35485): Not found: Tablet not found: c6bf0a83f20343afa47ead1c6e7dfd8d
I20260923 20:13:47.016259 18434 leader_election.cc:302] T c6bf0a83f20343afa47ead1c6e7dfd8d P da1071f8ba4c4d93b8a8d1d6f143babe [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: da1071f8ba4c4d93b8a8d1d6f143babe; no voters: 908ce75423b74581b7328602946ef436, b5dae6161ce245f8b7586278900d0d1e
I20260923 20:13:47.016728 18863 raft_consensus.cc:2726] T c6bf0a83f20343afa47ead1c6e7dfd8d P da1071f8ba4c4d93b8a8d1d6f143babe [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260923 20:13:47.410624 18864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:13:47.409323 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.190:44989 (local address 127.1.170.130:33077)
0923 20:13:47.410099 (+ 776us) negotiation.cc:107] Waiting for socket to connect
0923 20:13:47.410382 (+ 283us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.170.190:44989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":673,"thread_start_us":452,"threads_started":1}
I20260923 20:13:47.827752 1706 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskqUuku6/build/asan/bin/kudu pid 18375
I20260923 20:13:47.844900 18865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:13:47.829237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50496 (local address 127.1.170.129:43223)
0923 20:13:47.829712 (+ 475us) server_negotiation.cc:206] Beginning negotiation
0923 20:13:47.829724 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0923 20:13:47.829766 (+ 42us) server_negotiation.cc:407] Connection header received
0923 20:13:47.829852 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:13:47.829862 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:13:47.830201 (+ 339us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:13:47.830339 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:13:47.831998 (+ 1659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:13:47.833327 (+ 1329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:13:47.835087 (+ 1760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:13:47.835683 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:13:47.835799 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:13:47.838698 (+ 2899us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:13:47.838711 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:13:47.838728 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:13:47.838766 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:13:47.841709 (+ 2943us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:13:47.842518 (+ 809us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:13:47.842526 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:13:47.842528 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:13:47.842654 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:13:47.843050 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:13:47.843053 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:13:47.843055 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:13:47.843422 (+ 367us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:13:47.843928 (+ 506us) server_negotiation.cc:1035] Waiting for connection context
0923 20:13:47.844337 (+ 409us) server_negotiation.cc:299] Negotiation successful
0923 20:13:47.844586 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":320,"thread_start_us":275,"threads_started":1}
I20260923 20:13:47.846354 18541 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:13:47.854815 18867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:13:47.854050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.190:44989 (local address 127.1.170.131:41339)
0923 20:13:47.854550 (+ 500us) negotiation.cc:107] Waiting for socket to connect
0923 20:13:47.854664 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.170.190:44989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":387,"thread_start_us":262,"threads_started":1}
I20260923 20:13:47.946555 1706 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqUuku6/build/asan/bin/kudu with pid 18375
I20260923 20:13:47.964936 1706 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskqUuku6/build/asan/bin/kudu pid 18549
I20260923 20:13:47.980456 18868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:13:47.965839 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34808 (local address 127.1.170.130:35485)
0923 20:13:47.966547 (+ 708us) server_negotiation.cc:206] Beginning negotiation
0923 20:13:47.966552 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 20:13:47.966570 (+ 18us) server_negotiation.cc:407] Connection header received
0923 20:13:47.966626 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:13:47.966630 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:13:47.966689 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:13:47.966914 (+ 225us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:13:47.968231 (+ 1317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:13:47.969026 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:13:47.970794 (+ 1768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:13:47.971283 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:13:47.971390 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:13:47.974254 (+ 2864us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:13:47.974259 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:13:47.974263 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:13:47.974300 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:13:47.977222 (+ 2922us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:13:47.978404 (+ 1182us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:13:47.978408 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:13:47.978410 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:13:47.978567 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:13:47.978950 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:13:47.978954 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:13:47.978956 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:13:47.979243 (+ 287us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:13:47.979484 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0923 20:13:47.979804 (+ 320us) server_negotiation.cc:299] Negotiation successful
0923 20:13:47.980061 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":593,"thread_start_us":356,"threads_started":1}
I20260923 20:13:47.981144 18672 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:13:48.068761 1706 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqUuku6/build/asan/bin/kudu with pid 18549
I20260923 20:13:48.088582 1706 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskqUuku6/build/asan/bin/kudu pid 18680
I20260923 20:13:48.105700 18870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:13:48.089743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52924 (local address 127.1.170.131:34241)
0923 20:13:48.090457 (+ 714us) server_negotiation.cc:206] Beginning negotiation
0923 20:13:48.090462 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 20:13:48.090479 (+ 17us) server_negotiation.cc:407] Connection header received
0923 20:13:48.090534 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:13:48.090537 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:13:48.090611 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:13:48.090708 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:13:48.092131 (+ 1423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:13:48.092999 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:13:48.094463 (+ 1464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:13:48.094970 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:13:48.095081 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:13:48.098161 (+ 3080us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:13:48.098171 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:13:48.098176 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:13:48.098248 (+ 72us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:13:48.101140 (+ 2892us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:13:48.102709 (+ 1569us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:13:48.102716 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:13:48.102719 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:13:48.103002 (+ 283us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:13:48.103743 (+ 741us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:13:48.103748 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:13:48.103751 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:13:48.104140 (+ 389us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:13:48.104448 (+ 308us) server_negotiation.cc:1035] Waiting for connection context
0923 20:13:48.104972 (+ 524us) server_negotiation.cc:299] Negotiation successful
0923 20:13:48.105287 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":594,"thread_start_us":452,"threads_started":1}
I20260923 20:13:48.106739 18803 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:13:48.199904 1706 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqUuku6/build/asan/bin/kudu with pid 18680
2026-09-23T20:13:48Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (64921 ms)
[----------] 30 tests from SecurityITest (306559 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790194121577027-1706-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 20:13:48 dist-test-slave-th4g krb5kdc[18874](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 20:13:48 dist-test-slave-th4g krb5kdc[18874](info): set up 2 sockets
Sep 23 20:13:48 dist-test-slave-th4g krb5kdc[18874](info): commencing operation
krb5kdc: starting...
W20260923 20:13:49.346346 1706 mini_kdc.cc:121] Time spent starting KDC: real 1.103s user 0.001s sys 0.033s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790194121577027-1706-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790194121577027-1706-0/krb5kdc/test-user.keytab.
Sep 23 20:13:49 dist-test-slave-th4g krb5kdc[18874](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194429, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T20:13:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T20:13:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.170.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.170.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.170.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790194121577027-1706-0/krb5kdc/kudu_127.1.170.190.keytab.
Entry for principal kudu/127.1.170.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790194121577027-1706-0/krb5kdc/kudu_127.1.170.190.keytab.
I20260923 20:13:49.598917 1706 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqUuku6/build/asan/bin/kudu
/tmp/dist-test-taskqUuku6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790194121577027-1706-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790194121577027-1706-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790194121577027-1706-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790194121577027-1706-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.170.190:36963
--webserver_interface=127.1.170.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790194121577027-1706-0/krb5kdc/kudu_127.1.170.190.keytab
--principal=kudu/127.1.170.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.170.148:39879
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.170.190:36963
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790194121577027-1706-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790194121577027-1706-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790194121577027-1706-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:13:49.937963 18887 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:13:49.938269 18887 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:13:49.938364 18887 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:13:49.945609 18887 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 20:13:49.945714 18887 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:13:49.945760 18887 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:13:49.945827 18887 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 20:13:49.945870 18887 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 20:13:49.956334 18887 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.170.148:39879
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790194121577027-1706-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790194121577027-1706-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.170.190:36963
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790194121577027-1706-0/krb5kdc/kudu_127.1.170.190.keytab
--principal=kudu/127.1.170.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.170.190:36963
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790194121577027-1706-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.170.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790194121577027-1706-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:13:49.957623 18887 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:13:49.958900 18887 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:13:49.970513 18893 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:13:49.970726 18895 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:13:49.971045 18892 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:13:49.975644 18887 server_base.cc:1034] running on GCE node
I20260923 20:13:49.976401 18887 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:13:49.977856 18887 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:13:49.979254 18887 hybrid_clock.cc:648] HybridClock initialized: now 1790194429979155 us; error 51 us; skew 500 ppm
I20260923 20:13:49.979722 18887 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:13:49 dist-test-slave-th4g krb5kdc[18874](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194429, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.170.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:13:49.983773 18887 init.cc:377] Logged in from keytab as kudu/127.1.170.190@KRBTEST.COM (short username kudu)
I20260923 20:13:49.985316 18887 webserver.cc:460] Webserver started at http://127.1.170.190:40329/ using document root <none> and password file <none>
I20260923 20:13:49.986092 18887 fs_manager.cc:359] Metadata directory not provided
I20260923 20:13:49.986223 18887 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:13:49.986545 18887 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:13:49.989511 18887 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790194121577027-1706-0/minicluster-data/master-0/data/instance:
uuid: "aa87f9100b0c4ae3932ca34827cf796a"
format_stamp: "Formatted at 2026-09-23 20:13:49 on dist-test-slave-th4g"
server_key: "1563a47f47227ea91ff7cb948c0869bf"
server_key_iv: "a8108a30fe189ca1069c83943360a8d9"
server_key_version: "encryptionkey@0"
I20260923 20:13:49.990293 18887 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790194121577027-1706-0/minicluster-data/master-0/wal/instance:
uuid: "aa87f9100b0c4ae3932ca34827cf796a"
format_stamp: "Formatted at 2026-09-23 20:13:49 on dist-test-slave-th4g"
server_key: "1563a47f47227ea91ff7cb948c0869bf"
server_key_iv: "a8108a30fe189ca1069c83943360a8d9"
server_key_version: "encryptionkey@0"
I20260923 20:13:49.995558 18887 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260923 20:13:49.999084 18902 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:13:50.000532 18887 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260923 20:13:50.000734 18887 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790194121577027-1706-0/minicluster-data/master-0/data,/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790194121577027-1706-0/minicluster-data/master-0/wal
uuid: "aa87f9100b0c4ae3932ca34827cf796a"
format_stamp: "Formatted at 2026-09-23 20:13:49 on dist-test-slave-th4g"
server_key: "1563a47f47227ea91ff7cb948c0869bf"
server_key_iv: "a8108a30fe189ca1069c83943360a8d9"
server_key_version: "encryptionkey@0"
I20260923 20:13:50.000928 18887 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790194121577027-1706-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790194121577027-1706-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790194121577027-1706-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:13:50.013531 18887 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:13:50.018127 18887 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:13:50.018436 18887 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:13:50.038105 18887 rpc_server.cc:307] RPC server started. Bound to: 127.1.170.190:36963
I20260923 20:13:50.038128 18953 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.170.190:36963 every 8 connection(s)
I20260923 20:13:50.039499 18887 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790194121577027-1706-0/minicluster-data/master-0/data/info.pb
I20260923 20:13:50.042689 1706 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqUuku6/build/asan/bin/kudu as pid 18887
I20260923 20:13:50.042917 1706 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790194121577027-1706-0/minicluster-data/master-0/wal/instance
I20260923 20:13:50.043740 1706 external_mini_cluster.cc:1428] Setting key 3f498e556d08548335dde1bea6224395
I20260923 20:13:50.043967 18954 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:13:50.056604 18954 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aa87f9100b0c4ae3932ca34827cf796a: Bootstrap starting.
Sep 23 20:13:50 dist-test-slave-th4g krb5kdc[18874](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194429, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.170.190@KRBTEST.COM
I20260923 20:13:50.060297 18954 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P aa87f9100b0c4ae3932ca34827cf796a: Neither blocks nor log segments found. Creating new log.
I20260923 20:13:50.061427 18954 log.cc:824] T 00000000000000000000000000000000 P aa87f9100b0c4ae3932ca34827cf796a: Log is configured to *not* fsync() on all Append() calls
I20260923 20:13:50.064849 18954 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aa87f9100b0c4ae3932ca34827cf796a: No bootstrap required, opened a new log
I20260923 20:13:50.067346 18957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:13:50.045784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51360 (local address 127.1.170.190:36963)
0923 20:13:50.046587 (+ 803us) server_negotiation.cc:206] Beginning negotiation
0923 20:13:50.046599 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0923 20:13:50.046639 (+ 40us) server_negotiation.cc:407] Connection header received
0923 20:13:50.048736 (+ 2097us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:13:50.048757 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:13:50.049139 (+ 382us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:13:50.049522 (+ 383us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:13:50.051036 (+ 1514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:13:50.052279 (+ 1243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:13:50.053601 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:13:50.054456 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:13:50.054663 (+ 207us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:13:50.059341 (+ 4678us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:13:50.059349 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:13:50.059362 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:13:50.059403 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:13:50.063117 (+ 3714us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:13:50.063991 (+ 874us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:13:50.063997 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:13:50.064003 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:13:50.064154 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:13:50.064585 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:13:50.064588 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:13:50.064590 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:13:50.065147 (+ 557us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:13:50.065383 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0923 20:13:50.065975 (+ 592us) server_negotiation.cc:299] Negotiation successful
0923 20:13:50.066355 (+ 380us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":603,"thread_start_us":382,"threads_started":1}
I20260923 20:13:50.072944 18954 raft_consensus.cc:348] T 00000000000000000000000000000000 P aa87f9100b0c4ae3932ca34827cf796a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa87f9100b0c4ae3932ca34827cf796a" member_type: VOTER last_known_addr { host: "127.1.170.190" port: 36963 } }
I20260923 20:13:50.073350 18954 raft_consensus.cc:374] T 00000000000000000000000000000000 P aa87f9100b0c4ae3932ca34827cf796a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:13:50.073458 18954 raft_consensus.cc:729] T 00000000000000000000000000000000 P aa87f9100b0c4ae3932ca34827cf796a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aa87f9100b0c4ae3932ca34827cf796a, State: Initialized, Role: FOLLOWER
I20260923 20:13:50.074066 18954 consensus_queue.cc:260] T 00000000000000000000000000000000 P aa87f9100b0c4ae3932ca34827cf796a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa87f9100b0c4ae3932ca34827cf796a" member_type: VOTER last_known_addr { host: "127.1.170.190" port: 36963 } }
I20260923 20:13:50.074306 18954 raft_consensus.cc:388] T 00000000000000000000000000000000 P aa87f9100b0c4ae3932ca34827cf796a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 20:13:50.074450 18954 raft_consensus.cc:482] T 00000000000000000000000000000000 P aa87f9100b0c4ae3932ca34827cf796a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 20:13:50.074623 18954 raft_consensus.cc:3037] T 00000000000000000000000000000000 P aa87f9100b0c4ae3932ca34827cf796a [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:13:50.076792 18954 raft_consensus.cc:504] T 00000000000000000000000000000000 P aa87f9100b0c4ae3932ca34827cf796a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa87f9100b0c4ae3932ca34827cf796a" member_type: VOTER last_known_addr { host: "127.1.170.190" port: 36963 } }
I20260923 20:13:50.077301 18954 leader_election.cc:302] T 00000000000000000000000000000000 P aa87f9100b0c4ae3932ca34827cf796a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: aa87f9100b0c4ae3932ca34827cf796a; no voters:
I20260923 20:13:50.078019 18954 leader_election.cc:288] T 00000000000000000000000000000000 P aa87f9100b0c4ae3932ca34827cf796a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 20:13:50.078102 18959 raft_consensus.cc:2781] T 00000000000000000000000000000000 P aa87f9100b0c4ae3932ca34827cf796a [term 1 FOLLOWER]: Leader election won for term 1
I20260923 20:13:50.079068 18959 raft_consensus.cc:686] T 00000000000000000000000000000000 P aa87f9100b0c4ae3932ca34827cf796a [term 1 LEADER]: Becoming Leader. State: Replica: aa87f9100b0c4ae3932ca34827cf796a, State: Running, Role: LEADER
I20260923 20:13:50.079700 18959 consensus_queue.cc:237] T 00000000000000000000000000000000 P aa87f9100b0c4ae3932ca34827cf796a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa87f9100b0c4ae3932ca34827cf796a" member_type: VOTER last_known_addr { host: "127.1.170.190" port: 36963 } }
I20260923 20:13:50.080439 18954 sys_catalog.cc:564] T 00000000000000000000000000000000 P aa87f9100b0c4ae3932ca34827cf796a [sys.catalog]: configured and running, proceeding with master startup.
I20260923 20:13:50.082609 18961 sys_catalog.cc:455] T 00000000000000000000000000000000 P aa87f9100b0c4ae3932ca34827cf796a [sys.catalog]: SysCatalogTable state changed. Reason: New leader aa87f9100b0c4ae3932ca34827cf796a. Latest consensus state: current_term: 1 leader_uuid: "aa87f9100b0c4ae3932ca34827cf796a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa87f9100b0c4ae3932ca34827cf796a" member_type: VOTER last_known_addr { host: "127.1.170.190" port: 36963 } } }
I20260923 20:13:50.082818 18960 sys_catalog.cc:455] T 00000000000000000000000000000000 P aa87f9100b0c4ae3932ca34827cf796a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "aa87f9100b0c4ae3932ca34827cf796a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa87f9100b0c4ae3932ca34827cf796a" member_type: VOTER last_known_addr { host: "127.1.170.190" port: 36963 } } }
I20260923 20:13:50.083106 18961 sys_catalog.cc:458] T 00000000000000000000000000000000 P aa87f9100b0c4ae3932ca34827cf796a [sys.catalog]: This master's current role is: LEADER
I20260923 20:13:50.083165 18960 sys_catalog.cc:458] T 00000000000000000000000000000000 P aa87f9100b0c4ae3932ca34827cf796a [sys.catalog]: This master's current role is: LEADER
I20260923 20:13:50.085137 18964 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 20:13:50.091300 18964 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 20:13:50.100956 18964 catalog_manager.cc:1347] Generated new cluster ID: dc1944ff81314acaaa80baf92935f705
I20260923 20:13:50.101089 18964 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 20:13:50.118129 18964 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 20:13:50.119585 18964 catalog_manager.cc:1504] Loading token signing keys...
I20260923 20:13:50.129614 18964 catalog_manager.cc:5868] T 00000000000000000000000000000000 P aa87f9100b0c4ae3932ca34827cf796a: Generated new TSK 0
I20260923 20:13:50.130571 18964 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.170.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.170.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.170.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790194121577027-1706-0/krb5kdc/kudu_127.1.170.129.keytab.
Entry for principal kudu/127.1.170.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790194121577027-1706-0/krb5kdc/kudu_127.1.170.129.keytab.
I20260923 20:13:50.206533 1706 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqUuku6/build/asan/bin/kudu
/tmp/dist-test-taskqUuku6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790194121577027-1706-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790194121577027-1706-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790194121577027-1706-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790194121577027-1706-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.170.129:0
--local_ip_for_outbound_sockets=127.1.170.129
--webserver_interface=127.1.170.129
--webserver_port=0
--tserver_master_addrs=127.1.170.190:36963
--keytab_file=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790194121577027-1706-0/krb5kdc/kudu_127.1.170.129.keytab
--principal=kudu/127.1.170.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.170.148:39879
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790194121577027-1706-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790194121577027-1706-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790194121577027-1706-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:13:50.543852 18980 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:13:50.544149 18980 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:13:50.544322 18980 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:13:50.551668 18980 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:13:50.551833 18980 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:13:50.552011 18980 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.170.129
I20260923 20:13:50.561681 18980 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.170.148:39879
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790194121577027-1706-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790194121577027-1706-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790194121577027-1706-0/krb5kdc/kudu_127.1.170.129.keytab
--principal=kudu/127.1.170.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.170.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790194121577027-1706-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.170.129
--webserver_port=0
--tserver_master_addrs=127.1.170.190:36963
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.170.129
--log_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790194121577027-1706-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:13:50.562988 18980 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:13:50.564254 18980 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:13:50.577263 18985 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:13:50.577298 18986 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:13:50.577346 18988 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:13:50.578615 18980 server_base.cc:1034] running on GCE node
I20260923 20:13:50.579248 18980 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:13:50.580173 18980 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:13:50.581447 18980 hybrid_clock.cc:648] HybridClock initialized: now 1790194430581359 us; error 83 us; skew 500 ppm
I20260923 20:13:50.581844 18980 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:13:50 dist-test-slave-th4g krb5kdc[18874](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194430, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.170.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:13:50.584964 18980 init.cc:377] Logged in from keytab as kudu/127.1.170.129@KRBTEST.COM (short username kudu)
I20260923 20:13:50.586515 18980 webserver.cc:460] Webserver started at http://127.1.170.129:43881/ using document root <none> and password file <none>
I20260923 20:13:50.587177 18980 fs_manager.cc:359] Metadata directory not provided
I20260923 20:13:50.587276 18980 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:13:50.587594 18980 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:13:50.590504 18980 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790194121577027-1706-0/minicluster-data/ts-0/data/instance:
uuid: "dee43928c80f40da85ba7bb7bf49882c"
format_stamp: "Formatted at 2026-09-23 20:13:50 on dist-test-slave-th4g"
server_key: "31ddc11249c4b229a46779ba886f0ec7"
server_key_iv: "d477896e99ed6c55247dd36067a20adb"
server_key_version: "encryptionkey@0"
I20260923 20:13:50.591212 18980 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790194121577027-1706-0/minicluster-data/ts-0/wal/instance:
uuid: "dee43928c80f40da85ba7bb7bf49882c"
format_stamp: "Formatted at 2026-09-23 20:13:50 on dist-test-slave-th4g"
server_key: "31ddc11249c4b229a46779ba886f0ec7"
server_key_iv: "d477896e99ed6c55247dd36067a20adb"
server_key_version: "encryptionkey@0"
I20260923 20:13:50.596412 18980 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260923 20:13:50.599798 18995 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:13:50.601220 18980 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260923 20:13:50.601346 18980 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790194121577027-1706-0/minicluster-data/ts-0/data,/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790194121577027-1706-0/minicluster-data/ts-0/wal
uuid: "dee43928c80f40da85ba7bb7bf49882c"
format_stamp: "Formatted at 2026-09-23 20:13:50 on dist-test-slave-th4g"
server_key: "31ddc11249c4b229a46779ba886f0ec7"
server_key_iv: "d477896e99ed6c55247dd36067a20adb"
server_key_version: "encryptionkey@0"
I20260923 20:13:50.601512 18980 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790194121577027-1706-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790194121577027-1706-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790194121577027-1706-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:13:50.616786 18980 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:13:50.620710 18980 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:13:50.620955 18980 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:13:50.621861 18980 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 20:13:50.623278 18980 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 20:13:50.623353 18980 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 20:13:50.623432 18980 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 20:13:50.623481 18980 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 20:13:50.657796 18980 rpc_server.cc:307] RPC server started. Bound to: 127.1.170.129:32933
I20260923 20:13:50.657845 19107 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.170.129:32933 every 8 connection(s)
I20260923 20:13:50.659089 18980 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790194121577027-1706-0/minicluster-data/ts-0/data/info.pb
I20260923 20:13:50.663509 1706 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqUuku6/build/asan/bin/kudu as pid 18980
I20260923 20:13:50.663724 1706 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790194121577027-1706-0/minicluster-data/ts-0/wal/instance
I20260923 20:13:50.664353 1706 external_mini_cluster.cc:1428] Setting key 1bf7eb3863ee98038e4d5390a24524ed
Sep 23 20:13:50 dist-test-slave-th4g krb5kdc[18874](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194430, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.170.129@KRBTEST.COM for kudu/127.1.170.190@KRBTEST.COM
I20260923 20:13:50.678707 19110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:13:50.661486 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.170.129:38637 (local address 127.1.170.190:36963)
0923 20:13:50.662019 (+ 533us) server_negotiation.cc:206] Beginning negotiation
0923 20:13:50.662024 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 20:13:50.662716 (+ 692us) server_negotiation.cc:407] Connection header received
0923 20:13:50.664695 (+ 1979us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:13:50.664701 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:13:50.664790 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:13:50.664925 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:13:50.666715 (+ 1790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:13:50.667408 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:13:50.668959 (+ 1551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:13:50.669348 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:13:50.669487 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:13:50.673003 (+ 3516us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:13:50.673008 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:13:50.673012 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:13:50.673042 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:13:50.675592 (+ 2550us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:13:50.676324 (+ 732us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:13:50.676336 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:13:50.676338 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:13:50.676462 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:13:50.676933 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:13:50.676938 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:13:50.676946 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:13:50.677204 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:13:50.677377 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0923 20:13:50.678107 (+ 730us) server_negotiation.cc:299] Negotiation successful
0923 20:13:50.678341 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":415,"thread_start_us":270,"threads_started":1}
I20260923 20:13:50.679695 19111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:13:50.661818 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.190:36963 (local address 127.1.170.129:38637)
0923 20:13:50.662566 (+ 748us) negotiation.cc:107] Waiting for socket to connect
0923 20:13:50.662600 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0923 20:13:50.664424 (+ 1824us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:13:50.665413 (+ 989us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:13:50.665421 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:13:50.665972 (+ 551us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:13:50.666550 (+ 578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:13:50.666566 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:13:50.667587 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:13:50.667591 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:13:50.668802 (+ 1211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:13:50.668814 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:13:50.669573 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:13:50.669582 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:13:50.669834 (+ 252us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:13:50.670582 (+ 748us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:13:50.670605 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:13:50.672771 (+ 2166us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:13:50.675741 (+ 2970us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:13:50.675750 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:13:50.675767 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:13:50.676191 (+ 424us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:13:50.676587 (+ 396us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:13:50.676592 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:13:50.676596 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:13:50.676799 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:13:50.677335 (+ 536us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:13:50.677342 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:13:50.677691 (+ 349us) client_negotiation.cc:769] Sending connection context
0923 20:13:50.678002 (+ 311us) client_negotiation.cc:240] Negotiation successful
0923 20:13:50.678544 (+ 542us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":531,"thread_start_us":275,"threads_started":1}
I20260923 20:13:50.681339 19108 heartbeater.cc:347] Connected to a master server at 127.1.170.190:36963
I20260923 20:13:50.681655 19108 heartbeater.cc:464] Registering TS with master...
I20260923 20:13:50.682621 19108 heartbeater.cc:511] Master 127.1.170.190:36963 requested a full tablet report, sending...
I20260923 20:13:50.685249 18919 ts_manager.cc:194] Registered new tserver with Master: dee43928c80f40da85ba7bb7bf49882c (127.1.170.129:32933)
I20260923 20:13:50.687546 18919 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.170.129@KRBTEST.COM'} at 127.1.170.129:38637
WARNING: no policy specified for kudu/127.1.170.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.170.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.170.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790194121577027-1706-0/krb5kdc/kudu_127.1.170.130.keytab.
Entry for principal kudu/127.1.170.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790194121577027-1706-0/krb5kdc/kudu_127.1.170.130.keytab.
I20260923 20:13:50.722713 1706 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqUuku6/build/asan/bin/kudu
/tmp/dist-test-taskqUuku6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790194121577027-1706-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790194121577027-1706-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790194121577027-1706-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790194121577027-1706-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.170.130:0
--local_ip_for_outbound_sockets=127.1.170.130
--webserver_interface=127.1.170.130
--webserver_port=0
--tserver_master_addrs=127.1.170.190:36963
--keytab_file=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790194121577027-1706-0/krb5kdc/kudu_127.1.170.130.keytab
--principal=kudu/127.1.170.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.170.148:39879
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790194121577027-1706-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790194121577027-1706-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790194121577027-1706-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:13:51.064141 19114 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:13:51.064425 19114 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:13:51.064520 19114 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:13:51.071887 19114 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:13:51.072013 19114 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:13:51.072127 19114 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.170.130
I20260923 20:13:51.082286 19114 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.170.148:39879
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790194121577027-1706-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790194121577027-1706-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790194121577027-1706-0/krb5kdc/kudu_127.1.170.130.keytab
--principal=kudu/127.1.170.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.170.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790194121577027-1706-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.170.130
--webserver_port=0
--tserver_master_addrs=127.1.170.190:36963
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.170.130
--log_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790194121577027-1706-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:13:51.083576 19114 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:13:51.084964 19114 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:13:51.099073 19119 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:13:51.099378 19120 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:13:51.099993 19122 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:13:51.100303 19114 server_base.cc:1034] running on GCE node
I20260923 20:13:51.100994 19114 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:13:51.102131 19114 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:13:51.103400 19114 hybrid_clock.cc:648] HybridClock initialized: now 1790194431103344 us; error 44 us; skew 500 ppm
I20260923 20:13:51.103798 19114 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:13:51 dist-test-slave-th4g krb5kdc[18874](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194431, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.170.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:13:51.107515 19114 init.cc:377] Logged in from keytab as kudu/127.1.170.130@KRBTEST.COM (short username kudu)
I20260923 20:13:51.109278 19114 webserver.cc:460] Webserver started at http://127.1.170.130:45017/ using document root <none> and password file <none>
I20260923 20:13:51.110126 19114 fs_manager.cc:359] Metadata directory not provided
I20260923 20:13:51.110294 19114 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:13:51.110636 19114 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:13:51.113652 19114 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790194121577027-1706-0/minicluster-data/ts-1/data/instance:
uuid: "432510c38c6c4cac9557871bf41b4112"
format_stamp: "Formatted at 2026-09-23 20:13:51 on dist-test-slave-th4g"
server_key: "c2c454a81d7617dd2d33be296fe6d393"
server_key_iv: "7781b7b5f6583297455328cf04ad6391"
server_key_version: "encryptionkey@0"
I20260923 20:13:51.114514 19114 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790194121577027-1706-0/minicluster-data/ts-1/wal/instance:
uuid: "432510c38c6c4cac9557871bf41b4112"
format_stamp: "Formatted at 2026-09-23 20:13:51 on dist-test-slave-th4g"
server_key: "c2c454a81d7617dd2d33be296fe6d393"
server_key_iv: "7781b7b5f6583297455328cf04ad6391"
server_key_version: "encryptionkey@0"
I20260923 20:13:51.120448 19114 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260923 20:13:51.124995 19129 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:13:51.126978 19114 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.001s
I20260923 20:13:51.127207 19114 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790194121577027-1706-0/minicluster-data/ts-1/data,/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790194121577027-1706-0/minicluster-data/ts-1/wal
uuid: "432510c38c6c4cac9557871bf41b4112"
format_stamp: "Formatted at 2026-09-23 20:13:51 on dist-test-slave-th4g"
server_key: "c2c454a81d7617dd2d33be296fe6d393"
server_key_iv: "7781b7b5f6583297455328cf04ad6391"
server_key_version: "encryptionkey@0"
I20260923 20:13:51.127503 19114 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790194121577027-1706-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790194121577027-1706-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790194121577027-1706-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:13:51.149989 19114 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:13:51.154882 19114 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:13:51.155154 19114 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:13:51.156239 19114 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 20:13:51.157716 19114 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 20:13:51.157796 19114 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 20:13:51.157977 19114 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 20:13:51.158030 19114 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20260923 20:13:51.196606 19114 rpc_server.cc:307] RPC server started. Bound to: 127.1.170.130:40637
I20260923 20:13:51.196700 19241 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.170.130:40637 every 8 connection(s)
I20260923 20:13:51.198019 19114 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790194121577027-1706-0/minicluster-data/ts-1/data/info.pb
I20260923 20:13:51.202535 1706 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqUuku6/build/asan/bin/kudu as pid 19114
I20260923 20:13:51.202730 1706 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790194121577027-1706-0/minicluster-data/ts-1/wal/instance
I20260923 20:13:51.203347 1706 external_mini_cluster.cc:1428] Setting key e8ee7e82375c3df70719940345ccf9b9
Sep 23 20:13:51 dist-test-slave-th4g krb5kdc[18874](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194431, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.170.130@KRBTEST.COM for kudu/127.1.170.190@KRBTEST.COM
I20260923 20:13:51.222414 19244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:13:51.200959 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.170.130:32929 (local address 127.1.170.190:36963)
0923 20:13:51.201456 (+ 497us) server_negotiation.cc:206] Beginning negotiation
0923 20:13:51.201461 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 20:13:51.202336 (+ 875us) server_negotiation.cc:407] Connection header received
0923 20:13:51.204723 (+ 2387us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:13:51.204727 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:13:51.204796 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:13:51.204903 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:13:51.206927 (+ 2024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:13:51.208183 (+ 1256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:13:51.209942 (+ 1759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:13:51.210706 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:13:51.210859 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:13:51.215601 (+ 4742us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:13:51.215607 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:13:51.215611 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:13:51.215647 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:13:51.218513 (+ 2866us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:13:51.219609 (+ 1096us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:13:51.219613 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:13:51.219616 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:13:51.219768 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:13:51.220345 (+ 577us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:13:51.220349 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:13:51.220351 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:13:51.220659 (+ 308us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:13:51.220779 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0923 20:13:51.221803 (+ 1024us) server_negotiation.cc:299] Negotiation successful
0923 20:13:51.222130 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":378,"thread_start_us":269,"threads_started":1}
I20260923 20:13:51.223819 19245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:13:51.201318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.190:36963 (local address 127.1.170.130:32929)
0923 20:13:51.202174 (+ 856us) negotiation.cc:107] Waiting for socket to connect
0923 20:13:51.202225 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0923 20:13:51.204237 (+ 2012us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:13:51.205227 (+ 990us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:13:51.205238 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:13:51.206072 (+ 834us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:13:51.206737 (+ 665us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:13:51.206755 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:13:51.208381 (+ 1626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:13:51.208385 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:13:51.209726 (+ 1341us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:13:51.209741 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:13:51.210978 (+ 1237us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:13:51.210984 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:13:51.211193 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:13:51.211978 (+ 785us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:13:51.212004 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:13:51.215275 (+ 3271us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:13:51.218684 (+ 3409us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:13:51.218698 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:13:51.218718 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:13:51.219410 (+ 692us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:13:51.219907 (+ 497us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:13:51.219914 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:13:51.219921 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:13:51.220180 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:13:51.220797 (+ 617us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:13:51.220809 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:13:51.221353 (+ 544us) client_negotiation.cc:769] Sending connection context
0923 20:13:51.221673 (+ 320us) client_negotiation.cc:240] Negotiation successful
0923 20:13:51.222314 (+ 641us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":592,"thread_start_us":322,"threads_started":1}
I20260923 20:13:51.225739 19242 heartbeater.cc:347] Connected to a master server at 127.1.170.190:36963
I20260923 20:13:51.226119 19242 heartbeater.cc:464] Registering TS with master...
I20260923 20:13:51.227078 19242 heartbeater.cc:511] Master 127.1.170.190:36963 requested a full tablet report, sending...
I20260923 20:13:51.229269 18919 ts_manager.cc:194] Registered new tserver with Master: 432510c38c6c4cac9557871bf41b4112 (127.1.170.130:40637)
I20260923 20:13:51.230422 18919 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.170.130@KRBTEST.COM'} at 127.1.170.130:32929
WARNING: no policy specified for kudu/127.1.170.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.170.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.170.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790194121577027-1706-0/krb5kdc/kudu_127.1.170.131.keytab.
Entry for principal kudu/127.1.170.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790194121577027-1706-0/krb5kdc/kudu_127.1.170.131.keytab.
I20260923 20:13:51.270699 1706 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqUuku6/build/asan/bin/kudu
/tmp/dist-test-taskqUuku6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790194121577027-1706-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790194121577027-1706-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790194121577027-1706-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790194121577027-1706-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.170.131:0
--local_ip_for_outbound_sockets=127.1.170.131
--webserver_interface=127.1.170.131
--webserver_port=0
--tserver_master_addrs=127.1.170.190:36963
--keytab_file=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790194121577027-1706-0/krb5kdc/kudu_127.1.170.131.keytab
--principal=kudu/127.1.170.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.170.148:39879
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790194121577027-1706-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790194121577027-1706-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790194121577027-1706-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:13:51.612851 19248 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:13:51.613154 19248 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:13:51.613322 19248 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:13:51.622035 19248 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:13:51.622157 19248 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:13:51.622337 19248 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.170.131
I20260923 20:13:51.632751 19248 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.170.148:39879
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790194121577027-1706-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790194121577027-1706-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790194121577027-1706-0/krb5kdc/kudu_127.1.170.131.keytab
--principal=kudu/127.1.170.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.170.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790194121577027-1706-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.170.131
--webserver_port=0
--tserver_master_addrs=127.1.170.190:36963
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.170.131
--log_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790194121577027-1706-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:13:51.634236 19248 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:13:51.635555 19248 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:13:51.649204 19253 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:13:51.649132 19254 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:13:51.650132 19248 server_base.cc:1034] running on GCE node
W20260923 20:13:51.650261 19256 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:13:51.651162 19248 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:13:51.652494 19248 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:13:51.653812 19248 hybrid_clock.cc:648] HybridClock initialized: now 1790194431653715 us; error 66 us; skew 500 ppm
I20260923 20:13:51.654214 19248 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:13:51 dist-test-slave-th4g krb5kdc[18874](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194431, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.170.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:13:51.657748 19248 init.cc:377] Logged in from keytab as kudu/127.1.170.131@KRBTEST.COM (short username kudu)
I20260923 20:13:51.659439 19248 webserver.cc:460] Webserver started at http://127.1.170.131:33615/ using document root <none> and password file <none>
I20260923 20:13:51.660125 19248 fs_manager.cc:359] Metadata directory not provided
I20260923 20:13:51.660267 19248 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:13:51.660581 19248 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:13:51.663630 19248 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790194121577027-1706-0/minicluster-data/ts-2/data/instance:
uuid: "91fa880bb14b4b329f49778aa7320121"
format_stamp: "Formatted at 2026-09-23 20:13:51 on dist-test-slave-th4g"
server_key: "bdaa5c35d5c568067a85d88973ba78af"
server_key_iv: "209af220fced87b3e19715084b59ca50"
server_key_version: "encryptionkey@0"
I20260923 20:13:51.664438 19248 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790194121577027-1706-0/minicluster-data/ts-2/wal/instance:
uuid: "91fa880bb14b4b329f49778aa7320121"
format_stamp: "Formatted at 2026-09-23 20:13:51 on dist-test-slave-th4g"
server_key: "bdaa5c35d5c568067a85d88973ba78af"
server_key_iv: "209af220fced87b3e19715084b59ca50"
server_key_version: "encryptionkey@0"
I20260923 20:13:51.669845 19248 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260923 20:13:51.673648 19263 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:13:51.675071 19248 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260923 20:13:51.675266 19248 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790194121577027-1706-0/minicluster-data/ts-2/data,/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790194121577027-1706-0/minicluster-data/ts-2/wal
uuid: "91fa880bb14b4b329f49778aa7320121"
format_stamp: "Formatted at 2026-09-23 20:13:51 on dist-test-slave-th4g"
server_key: "bdaa5c35d5c568067a85d88973ba78af"
server_key_iv: "209af220fced87b3e19715084b59ca50"
server_key_version: "encryptionkey@0"
I20260923 20:13:51.675487 19248 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790194121577027-1706-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790194121577027-1706-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790194121577027-1706-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:13:51.691035 19108 heartbeater.cc:503] Master 127.1.170.190:36963 was elected leader, sending a full tablet report...
I20260923 20:13:51.691092 19248 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:13:51.695907 19248 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:13:51.696151 19248 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:13:51.697160 19248 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 20:13:51.698441 19248 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 20:13:51.698515 19248 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 20:13:51.698596 19248 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 20:13:51.698643 19248 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 20:13:51.735036 19248 rpc_server.cc:307] RPC server started. Bound to: 127.1.170.131:36237
I20260923 20:13:51.735081 19375 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.170.131:36237 every 8 connection(s)
I20260923 20:13:51.736351 19248 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790194121577027-1706-0/minicluster-data/ts-2/data/info.pb
I20260923 20:13:51.737511 1706 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqUuku6/build/asan/bin/kudu as pid 19248
I20260923 20:13:51.737699 1706 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790194121577027-1706-0/minicluster-data/ts-2/wal/instance
I20260923 20:13:51.738263 1706 external_mini_cluster.cc:1428] Setting key 9780761fffef422c50aff2a359905285
Sep 23 20:13:51 dist-test-slave-th4g krb5kdc[18874](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194431, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.170.131@KRBTEST.COM for kudu/127.1.170.190@KRBTEST.COM
I20260923 20:13:51.768389 19378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:13:51.739440 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.170.131:48807 (local address 127.1.170.190:36963)
0923 20:13:51.739948 (+ 508us) server_negotiation.cc:206] Beginning negotiation
0923 20:13:51.739952 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 20:13:51.741376 (+ 1424us) server_negotiation.cc:407] Connection header received
0923 20:13:51.743857 (+ 2481us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:13:51.743862 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:13:51.743960 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:13:51.744096 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:13:51.745683 (+ 1587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:13:51.746822 (+ 1139us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:13:51.748560 (+ 1738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:13:51.748970 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:13:51.749108 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:13:51.759664 (+ 10556us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:13:51.759670 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:13:51.759675 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:13:51.759717 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:13:51.763178 (+ 3461us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:13:51.764266 (+ 1088us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:13:51.764271 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:13:51.764276 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:13:51.764432 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:13:51.764978 (+ 546us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:13:51.764983 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:13:51.764986 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:13:51.765896 (+ 910us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:13:51.766138 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0923 20:13:51.767558 (+ 1420us) server_negotiation.cc:299] Negotiation successful
0923 20:13:51.767947 (+ 389us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":376,"thread_start_us":253,"threads_started":1}
I20260923 20:13:51.769245 19379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:13:51.740130 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.190:36963 (local address 127.1.170.131:48807)
0923 20:13:51.741121 (+ 991us) negotiation.cc:107] Waiting for socket to connect
0923 20:13:51.741163 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0923 20:13:51.743514 (+ 2351us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:13:51.744348 (+ 834us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:13:51.744358 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:13:51.744901 (+ 543us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:13:51.745489 (+ 588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:13:51.745507 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:13:51.747060 (+ 1553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:13:51.747070 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:13:51.748382 (+ 1312us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:13:51.748395 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:13:51.749218 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:13:51.749225 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:13:51.749483 (+ 258us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:13:51.755706 (+ 6223us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:13:51.755748 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:13:51.759347 (+ 3599us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:13:51.763403 (+ 4056us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:13:51.763415 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:13:51.763441 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:13:51.764080 (+ 639us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:13:51.764568 (+ 488us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:13:51.764575 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:13:51.764578 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:13:51.764845 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:13:51.766199 (+ 1354us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:13:51.766207 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:13:51.766733 (+ 526us) client_negotiation.cc:769] Sending connection context
0923 20:13:51.767031 (+ 298us) client_negotiation.cc:240] Negotiation successful
0923 20:13:51.767590 (+ 559us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":773,"thread_start_us":598,"threads_started":1}
I20260923 20:13:51.770618 19376 heartbeater.cc:347] Connected to a master server at 127.1.170.190:36963
I20260923 20:13:51.770908 19376 heartbeater.cc:464] Registering TS with master...
I20260923 20:13:51.771590 19376 heartbeater.cc:511] Master 127.1.170.190:36963 requested a full tablet report, sending...
I20260923 20:13:51.773128 18919 ts_manager.cc:194] Registered new tserver with Master: 91fa880bb14b4b329f49778aa7320121 (127.1.170.131:36237)
I20260923 20:13:51.774217 18919 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.170.131@KRBTEST.COM'} at 127.1.170.131:48807
I20260923 20:13:51.782388 1706 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260923 20:13:51.800455 19378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:13:51.787134 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51372 (local address 127.1.170.190:36963)
0923 20:13:51.787374 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0923 20:13:51.787383 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 20:13:51.787767 (+ 384us) server_negotiation.cc:407] Connection header received
0923 20:13:51.787972 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:13:51.787976 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:13:51.788055 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:13:51.788138 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:13:51.789540 (+ 1402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:13:51.790401 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:13:51.791693 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:13:51.792103 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:13:51.792206 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:13:51.794815 (+ 2609us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:13:51.794820 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:13:51.794823 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:13:51.794867 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:13:51.797573 (+ 2706us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:13:51.798400 (+ 827us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:13:51.798404 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:13:51.798416 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:13:51.798557 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:13:51.798926 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:13:51.798929 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:13:51.798931 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:13:51.799268 (+ 337us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:13:51.799452 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0923 20:13:51.799903 (+ 451us) server_negotiation.cc:299] Negotiation successful
0923 20:13:51.800088 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260923 20:13:51.805330 18919 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51372:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 20:13:51.812367 18919 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 20:13:51.847528 19389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:13:51.825078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.130:40637 (local address 127.0.0.1:55550)
0923 20:13:51.837532 (+ 12454us) negotiation.cc:107] Waiting for socket to connect
0923 20:13:51.837597 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0923 20:13:51.837820 (+ 223us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:13:51.839027 (+ 1207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:13:51.839037 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:13:51.839070 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:13:51.839548 (+ 478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:13:51.839560 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:13:51.842027 (+ 2467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:13:51.842033 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:13:51.844582 (+ 2549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:13:51.844606 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:13:51.846392 (+ 1786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:13:51.846398 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:13:51.846601 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:13:51.846658 (+ 57us) client_negotiation.cc:769] Sending connection context
0923 20:13:51.846775 (+ 117us) client_negotiation.cc:240] Negotiation successful
0923 20:13:51.846992 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12143,"mutex_wait_us":6,"thread_start_us":11993,"threads_started":1}
I20260923 20:13:51.848043 19392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:13:51.825367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55550 (local address 127.1.170.130:40637)
0923 20:13:51.838478 (+ 13111us) server_negotiation.cc:206] Beginning negotiation
0923 20:13:51.838498 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0923 20:13:51.838522 (+ 24us) server_negotiation.cc:407] Connection header received
0923 20:13:51.838602 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:13:51.838608 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:13:51.838789 (+ 181us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:13:51.839750 (+ 961us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:13:51.840020 (+ 270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:13:51.841832 (+ 1812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:13:51.844819 (+ 2987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:13:51.846171 (+ 1352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:13:51.847202 (+ 1031us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:13:51.847406 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0923 20:13:51.847485 (+ 79us) server_negotiation.cc:299] Negotiation successful
0923 20:13:51.847676 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12949,"thread_start_us":12781,"threads_started":1}
I20260923 20:13:51.848477 19390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:13:51.825164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56840 (local address 127.1.170.129:32933)
0923 20:13:51.837032 (+ 11868us) server_negotiation.cc:206] Beginning negotiation
0923 20:13:51.837045 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0923 20:13:51.838302 (+ 1257us) server_negotiation.cc:407] Connection header received
0923 20:13:51.838426 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:13:51.838434 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:13:51.838734 (+ 300us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:13:51.838961 (+ 227us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:13:51.840482 (+ 1521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:13:51.843686 (+ 3204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:13:51.845586 (+ 1900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:13:51.847057 (+ 1471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:13:51.847178 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:13:51.847401 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0923 20:13:51.848064 (+ 663us) server_negotiation.cc:299] Negotiation successful
0923 20:13:51.848232 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11656,"thread_start_us":11487,"threads_started":1}
I20260923 20:13:51.848981 19394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:13:51.827819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50620 (local address 127.1.170.131:36237)
0923 20:13:51.839541 (+ 11722us) server_negotiation.cc:206] Beginning negotiation
0923 20:13:51.839549 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 20:13:51.839853 (+ 304us) server_negotiation.cc:407] Connection header received
0923 20:13:51.840083 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:13:51.840092 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:13:51.840424 (+ 332us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:13:51.840646 (+ 222us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:13:51.841236 (+ 590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:13:51.842882 (+ 1646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:13:51.845415 (+ 2533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:13:51.846865 (+ 1450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:13:51.846996 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:13:51.847245 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0923 20:13:51.848502 (+ 1257us) server_negotiation.cc:299] Negotiation successful
0923 20:13:51.848657 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11449,"thread_start_us":11257,"threads_started":1}
I20260923 20:13:51.849108 19393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:13:51.827078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.131:36237 (local address 127.0.0.1:50620)
0923 20:13:51.839724 (+ 12646us) negotiation.cc:107] Waiting for socket to connect
0923 20:13:51.839741 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0923 20:13:51.839876 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:13:51.840639 (+ 763us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:13:51.840647 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:13:51.840676 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:13:51.841043 (+ 367us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:13:51.841054 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:13:51.843054 (+ 2000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:13:51.843059 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:13:51.845226 (+ 2167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:13:51.845244 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:13:51.847412 (+ 2168us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:13:51.847420 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:13:51.847904 (+ 484us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:13:51.847932 (+ 28us) client_negotiation.cc:769] Sending connection context
0923 20:13:51.848001 (+ 69us) client_negotiation.cc:240] Negotiation successful
0923 20:13:51.848797 (+ 796us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12510,"thread_start_us":12600,"threads_started":1}
I20260923 20:13:51.849298 19391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:13:51.825125 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.129:32933 (local address 127.0.0.1:56840)
0923 20:13:51.837532 (+ 12407us) negotiation.cc:107] Waiting for socket to connect
0923 20:13:51.838035 (+ 503us) client_negotiation.cc:174] Beginning negotiation
0923 20:13:51.838167 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:13:51.839221 (+ 1054us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:13:51.839225 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:13:51.839244 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:13:51.840233 (+ 989us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:13:51.840243 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:13:51.843873 (+ 3630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:13:51.843877 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:13:51.845394 (+ 1517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:13:51.845409 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:13:51.847545 (+ 2136us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:13:51.847551 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:13:51.847713 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:13:51.847807 (+ 94us) client_negotiation.cc:769] Sending connection context
0923 20:13:51.847893 (+ 86us) client_negotiation.cc:240] Negotiation successful
0923 20:13:51.848031 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12227,"mutex_wait_us":649,"thread_start_us":12386,"threads_started":1}
I20260923 20:13:51.852546 19177 tablet_service.cc:1467] Processing CreateTablet for tablet e433fc8d3d0f4107a3da1eb05040f2e1 (DEFAULT_TABLE table=test-table [id=edd9079660fb4828b9f74527c18e2790]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 20:13:51.852815 19043 tablet_service.cc:1467] Processing CreateTablet for tablet e433fc8d3d0f4107a3da1eb05040f2e1 (DEFAULT_TABLE table=test-table [id=edd9079660fb4828b9f74527c18e2790]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 20:13:51.852779 19311 tablet_service.cc:1467] Processing CreateTablet for tablet e433fc8d3d0f4107a3da1eb05040f2e1 (DEFAULT_TABLE table=test-table [id=edd9079660fb4828b9f74527c18e2790]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 20:13:51.853835 19177 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e433fc8d3d0f4107a3da1eb05040f2e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:13:51.854089 19043 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e433fc8d3d0f4107a3da1eb05040f2e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:13:51.854105 19311 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e433fc8d3d0f4107a3da1eb05040f2e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:13:51.865278 19395 tablet_bootstrap.cc:492] T e433fc8d3d0f4107a3da1eb05040f2e1 P 432510c38c6c4cac9557871bf41b4112: Bootstrap starting.
I20260923 20:13:51.868254 19396 tablet_bootstrap.cc:492] T e433fc8d3d0f4107a3da1eb05040f2e1 P dee43928c80f40da85ba7bb7bf49882c: Bootstrap starting.
I20260923 20:13:51.870004 19395 tablet_bootstrap.cc:654] T e433fc8d3d0f4107a3da1eb05040f2e1 P 432510c38c6c4cac9557871bf41b4112: Neither blocks nor log segments found. Creating new log.
I20260923 20:13:51.870584 19397 tablet_bootstrap.cc:492] T e433fc8d3d0f4107a3da1eb05040f2e1 P 91fa880bb14b4b329f49778aa7320121: Bootstrap starting.
I20260923 20:13:51.871493 19395 log.cc:824] T e433fc8d3d0f4107a3da1eb05040f2e1 P 432510c38c6c4cac9557871bf41b4112: Log is configured to *not* fsync() on all Append() calls
I20260923 20:13:51.871958 19396 tablet_bootstrap.cc:654] T e433fc8d3d0f4107a3da1eb05040f2e1 P dee43928c80f40da85ba7bb7bf49882c: Neither blocks nor log segments found. Creating new log.
I20260923 20:13:51.873081 19396 log.cc:824] T e433fc8d3d0f4107a3da1eb05040f2e1 P dee43928c80f40da85ba7bb7bf49882c: Log is configured to *not* fsync() on all Append() calls
I20260923 20:13:51.873953 19397 tablet_bootstrap.cc:654] T e433fc8d3d0f4107a3da1eb05040f2e1 P 91fa880bb14b4b329f49778aa7320121: Neither blocks nor log segments found. Creating new log.
I20260923 20:13:51.874871 19395 tablet_bootstrap.cc:492] T e433fc8d3d0f4107a3da1eb05040f2e1 P 432510c38c6c4cac9557871bf41b4112: No bootstrap required, opened a new log
I20260923 20:13:51.875150 19395 ts_tablet_manager.cc:1397] T e433fc8d3d0f4107a3da1eb05040f2e1 P 432510c38c6c4cac9557871bf41b4112: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.007s
I20260923 20:13:51.875138 19397 log.cc:824] T e433fc8d3d0f4107a3da1eb05040f2e1 P 91fa880bb14b4b329f49778aa7320121: Log is configured to *not* fsync() on all Append() calls
I20260923 20:13:51.876130 19396 tablet_bootstrap.cc:492] T e433fc8d3d0f4107a3da1eb05040f2e1 P dee43928c80f40da85ba7bb7bf49882c: No bootstrap required, opened a new log
I20260923 20:13:51.876408 19396 ts_tablet_manager.cc:1397] T e433fc8d3d0f4107a3da1eb05040f2e1 P dee43928c80f40da85ba7bb7bf49882c: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260923 20:13:51.886457 19397 tablet_bootstrap.cc:492] T e433fc8d3d0f4107a3da1eb05040f2e1 P 91fa880bb14b4b329f49778aa7320121: No bootstrap required, opened a new log
I20260923 20:13:51.886804 19397 ts_tablet_manager.cc:1397] T e433fc8d3d0f4107a3da1eb05040f2e1 P 91fa880bb14b4b329f49778aa7320121: Time spent bootstrapping tablet: real 0.016s user 0.006s sys 0.009s
I20260923 20:13:51.886831 19395 raft_consensus.cc:348] T e433fc8d3d0f4107a3da1eb05040f2e1 P 432510c38c6c4cac9557871bf41b4112 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dee43928c80f40da85ba7bb7bf49882c" member_type: VOTER last_known_addr { host: "127.1.170.129" port: 32933 } } peers { permanent_uuid: "432510c38c6c4cac9557871bf41b4112" member_type: VOTER last_known_addr { host: "127.1.170.130" port: 40637 } } peers { permanent_uuid: "91fa880bb14b4b329f49778aa7320121" member_type: VOTER last_known_addr { host: "127.1.170.131" port: 36237 } }
I20260923 20:13:51.887378 19395 raft_consensus.cc:374] T e433fc8d3d0f4107a3da1eb05040f2e1 P 432510c38c6c4cac9557871bf41b4112 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:13:51.887482 19395 raft_consensus.cc:729] T e433fc8d3d0f4107a3da1eb05040f2e1 P 432510c38c6c4cac9557871bf41b4112 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 432510c38c6c4cac9557871bf41b4112, State: Initialized, Role: FOLLOWER
I20260923 20:13:51.887223 19396 raft_consensus.cc:348] T e433fc8d3d0f4107a3da1eb05040f2e1 P dee43928c80f40da85ba7bb7bf49882c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dee43928c80f40da85ba7bb7bf49882c" member_type: VOTER last_known_addr { host: "127.1.170.129" port: 32933 } } peers { permanent_uuid: "432510c38c6c4cac9557871bf41b4112" member_type: VOTER last_known_addr { host: "127.1.170.130" port: 40637 } } peers { permanent_uuid: "91fa880bb14b4b329f49778aa7320121" member_type: VOTER last_known_addr { host: "127.1.170.131" port: 36237 } }
I20260923 20:13:51.887656 19396 raft_consensus.cc:374] T e433fc8d3d0f4107a3da1eb05040f2e1 P dee43928c80f40da85ba7bb7bf49882c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:13:51.887751 19396 raft_consensus.cc:729] T e433fc8d3d0f4107a3da1eb05040f2e1 P dee43928c80f40da85ba7bb7bf49882c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dee43928c80f40da85ba7bb7bf49882c, State: Initialized, Role: FOLLOWER
I20260923 20:13:51.888303 19396 consensus_queue.cc:260] T e433fc8d3d0f4107a3da1eb05040f2e1 P dee43928c80f40da85ba7bb7bf49882c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dee43928c80f40da85ba7bb7bf49882c" member_type: VOTER last_known_addr { host: "127.1.170.129" port: 32933 } } peers { permanent_uuid: "432510c38c6c4cac9557871bf41b4112" member_type: VOTER last_known_addr { host: "127.1.170.130" port: 40637 } } peers { permanent_uuid: "91fa880bb14b4b329f49778aa7320121" member_type: VOTER last_known_addr { host: "127.1.170.131" port: 36237 } }
I20260923 20:13:51.888289 19395 consensus_queue.cc:260] T e433fc8d3d0f4107a3da1eb05040f2e1 P 432510c38c6c4cac9557871bf41b4112 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dee43928c80f40da85ba7bb7bf49882c" member_type: VOTER last_known_addr { host: "127.1.170.129" port: 32933 } } peers { permanent_uuid: "432510c38c6c4cac9557871bf41b4112" member_type: VOTER last_known_addr { host: "127.1.170.130" port: 40637 } } peers { permanent_uuid: "91fa880bb14b4b329f49778aa7320121" member_type: VOTER last_known_addr { host: "127.1.170.131" port: 36237 } }
I20260923 20:13:51.895062 19397 raft_consensus.cc:348] T e433fc8d3d0f4107a3da1eb05040f2e1 P 91fa880bb14b4b329f49778aa7320121 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dee43928c80f40da85ba7bb7bf49882c" member_type: VOTER last_known_addr { host: "127.1.170.129" port: 32933 } } peers { permanent_uuid: "432510c38c6c4cac9557871bf41b4112" member_type: VOTER last_known_addr { host: "127.1.170.130" port: 40637 } } peers { permanent_uuid: "91fa880bb14b4b329f49778aa7320121" member_type: VOTER last_known_addr { host: "127.1.170.131" port: 36237 } }
I20260923 20:13:51.895534 19242 heartbeater.cc:503] Master 127.1.170.190:36963 was elected leader, sending a full tablet report...
I20260923 20:13:51.895543 19397 raft_consensus.cc:374] T e433fc8d3d0f4107a3da1eb05040f2e1 P 91fa880bb14b4b329f49778aa7320121 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:13:51.895648 19397 raft_consensus.cc:729] T e433fc8d3d0f4107a3da1eb05040f2e1 P 91fa880bb14b4b329f49778aa7320121 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 91fa880bb14b4b329f49778aa7320121, State: Initialized, Role: FOLLOWER
I20260923 20:13:51.896638 19396 ts_tablet_manager.cc:1428] T e433fc8d3d0f4107a3da1eb05040f2e1 P dee43928c80f40da85ba7bb7bf49882c: Time spent starting tablet: real 0.020s user 0.016s sys 0.004s
I20260923 20:13:51.897400 19395 ts_tablet_manager.cc:1428] T e433fc8d3d0f4107a3da1eb05040f2e1 P 432510c38c6c4cac9557871bf41b4112: Time spent starting tablet: real 0.022s user 0.010s sys 0.011s
I20260923 20:13:51.897730 19397 consensus_queue.cc:260] T e433fc8d3d0f4107a3da1eb05040f2e1 P 91fa880bb14b4b329f49778aa7320121 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dee43928c80f40da85ba7bb7bf49882c" member_type: VOTER last_known_addr { host: "127.1.170.129" port: 32933 } } peers { permanent_uuid: "432510c38c6c4cac9557871bf41b4112" member_type: VOTER last_known_addr { host: "127.1.170.130" port: 40637 } } peers { permanent_uuid: "91fa880bb14b4b329f49778aa7320121" member_type: VOTER last_known_addr { host: "127.1.170.131" port: 36237 } }
I20260923 20:13:51.899344 19376 heartbeater.cc:503] Master 127.1.170.190:36963 was elected leader, sending a full tablet report...
I20260923 20:13:51.899636 19397 ts_tablet_manager.cc:1428] T e433fc8d3d0f4107a3da1eb05040f2e1 P 91fa880bb14b4b329f49778aa7320121: Time spent starting tablet: real 0.013s user 0.011s sys 0.000s
W20260923 20:13:51.911532 19109 tablet.cc:2369] T e433fc8d3d0f4107a3da1eb05040f2e1 P dee43928c80f40da85ba7bb7bf49882c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260923 20:13:51.950515 19243 tablet.cc:2369] T e433fc8d3d0f4107a3da1eb05040f2e1 P 432510c38c6c4cac9557871bf41b4112: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260923 20:13:51.988123 19377 tablet.cc:2369] T e433fc8d3d0f4107a3da1eb05040f2e1 P 91fa880bb14b4b329f49778aa7320121: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 20:13:52.205415 19401 raft_consensus.cc:482] T e433fc8d3d0f4107a3da1eb05040f2e1 P dee43928c80f40da85ba7bb7bf49882c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 20:13:52.205770 19401 raft_consensus.cc:504] T e433fc8d3d0f4107a3da1eb05040f2e1 P dee43928c80f40da85ba7bb7bf49882c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dee43928c80f40da85ba7bb7bf49882c" member_type: VOTER last_known_addr { host: "127.1.170.129" port: 32933 } } peers { permanent_uuid: "432510c38c6c4cac9557871bf41b4112" member_type: VOTER last_known_addr { host: "127.1.170.130" port: 40637 } } peers { permanent_uuid: "91fa880bb14b4b329f49778aa7320121" member_type: VOTER last_known_addr { host: "127.1.170.131" port: 36237 } }
I20260923 20:13:52.207978 19401 leader_election.cc:288] T e433fc8d3d0f4107a3da1eb05040f2e1 P dee43928c80f40da85ba7bb7bf49882c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 432510c38c6c4cac9557871bf41b4112 (127.1.170.130:40637), 91fa880bb14b4b329f49778aa7320121 (127.1.170.131:36237)
I20260923 20:13:52.218446 19392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:13:52.208955 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.170.129:46981 (local address 127.1.170.130:40637)
0923 20:13:52.209180 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0923 20:13:52.209188 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 20:13:52.209822 (+ 634us) server_negotiation.cc:407] Connection header received
0923 20:13:52.210093 (+ 271us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:13:52.210099 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:13:52.210175 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:13:52.210274 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:13:52.211182 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:13:52.212295 (+ 1113us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:13:52.215875 (+ 3580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:13:52.217152 (+ 1277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:13:52.217324 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:13:52.217402 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0923 20:13:52.218034 (+ 632us) server_negotiation.cc:299] Negotiation successful
0923 20:13:52.218172 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260923 20:13:52.218453 19394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:13:52.208678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.170.129:46857 (local address 127.1.170.131:36237)
0923 20:13:52.208944 (+ 266us) server_negotiation.cc:206] Beginning negotiation
0923 20:13:52.208954 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0923 20:13:52.209820 (+ 866us) server_negotiation.cc:407] Connection header received
0923 20:13:52.210093 (+ 273us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:13:52.210099 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:13:52.210175 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:13:52.210274 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:13:52.211182 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:13:52.212302 (+ 1120us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:13:52.215875 (+ 3573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:13:52.217152 (+ 1277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:13:52.217359 (+ 207us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:13:52.217428 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0923 20:13:52.218034 (+ 606us) server_negotiation.cc:299] Negotiation successful
0923 20:13:52.218180 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260923 20:13:52.218676 19405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:13:52.208216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.131:36237 (local address 127.1.170.129:46857)
0923 20:13:52.209672 (+ 1456us) negotiation.cc:107] Waiting for socket to connect
0923 20:13:52.209714 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0923 20:13:52.209900 (+ 186us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:13:52.210326 (+ 426us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:13:52.210332 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:13:52.210380 (+ 48us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:13:52.210992 (+ 612us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:13:52.211002 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:13:52.212476 (+ 1474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:13:52.212480 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:13:52.215605 (+ 3125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:13:52.215624 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:13:52.217486 (+ 1862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:13:52.217493 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:13:52.217764 (+ 271us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:13:52.217861 (+ 97us) client_negotiation.cc:769] Sending connection context
0923 20:13:52.217990 (+ 129us) client_negotiation.cc:240] Negotiation successful
0923 20:13:52.218183 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1291,"mutex_wait_us":413,"spinlock_wait_cycles":11392,"thread_start_us":1176,"threads_started":1}
I20260923 20:13:52.219008 19404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:13:52.208344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.130:40637 (local address 127.1.170.129:46981)
0923 20:13:52.209695 (+ 1351us) negotiation.cc:107] Waiting for socket to connect
0923 20:13:52.209713 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0923 20:13:52.209953 (+ 240us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:13:52.210326 (+ 373us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:13:52.210332 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:13:52.210382 (+ 50us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:13:52.210977 (+ 595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:13:52.210994 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:13:52.212474 (+ 1480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:13:52.212480 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:13:52.215605 (+ 3125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:13:52.215624 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:13:52.217498 (+ 1874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:13:52.217503 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:13:52.217802 (+ 299us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:13:52.217863 (+ 61us) client_negotiation.cc:769] Sending connection context
0923 20:13:52.217990 (+ 127us) client_negotiation.cc:240] Negotiation successful
0923 20:13:52.218183 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1246,"mutex_wait_us":345,"spinlock_wait_cycles":42112,"thread_start_us":879,"threads_started":1}
I20260923 20:13:52.219831 19331 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e433fc8d3d0f4107a3da1eb05040f2e1" candidate_uuid: "dee43928c80f40da85ba7bb7bf49882c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "91fa880bb14b4b329f49778aa7320121" is_pre_election: true
I20260923 20:13:52.220145 19197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e433fc8d3d0f4107a3da1eb05040f2e1" candidate_uuid: "dee43928c80f40da85ba7bb7bf49882c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "432510c38c6c4cac9557871bf41b4112" is_pre_election: true
I20260923 20:13:52.220311 19331 raft_consensus.cc:2454] T e433fc8d3d0f4107a3da1eb05040f2e1 P 91fa880bb14b4b329f49778aa7320121 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dee43928c80f40da85ba7bb7bf49882c in term 0.
I20260923 20:13:52.220614 19197 raft_consensus.cc:2454] T e433fc8d3d0f4107a3da1eb05040f2e1 P 432510c38c6c4cac9557871bf41b4112 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dee43928c80f40da85ba7bb7bf49882c in term 0.
I20260923 20:13:52.220983 18996 leader_election.cc:302] T e433fc8d3d0f4107a3da1eb05040f2e1 P dee43928c80f40da85ba7bb7bf49882c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 91fa880bb14b4b329f49778aa7320121, dee43928c80f40da85ba7bb7bf49882c; no voters:
I20260923 20:13:52.221452 19401 raft_consensus.cc:2781] T e433fc8d3d0f4107a3da1eb05040f2e1 P dee43928c80f40da85ba7bb7bf49882c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 20:13:52.221601 19401 raft_consensus.cc:482] T e433fc8d3d0f4107a3da1eb05040f2e1 P dee43928c80f40da85ba7bb7bf49882c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 20:13:52.221717 19401 raft_consensus.cc:3037] T e433fc8d3d0f4107a3da1eb05040f2e1 P dee43928c80f40da85ba7bb7bf49882c [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:13:52.223875 19401 raft_consensus.cc:504] T e433fc8d3d0f4107a3da1eb05040f2e1 P dee43928c80f40da85ba7bb7bf49882c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dee43928c80f40da85ba7bb7bf49882c" member_type: VOTER last_known_addr { host: "127.1.170.129" port: 32933 } } peers { permanent_uuid: "432510c38c6c4cac9557871bf41b4112" member_type: VOTER last_known_addr { host: "127.1.170.130" port: 40637 } } peers { permanent_uuid: "91fa880bb14b4b329f49778aa7320121" member_type: VOTER last_known_addr { host: "127.1.170.131" port: 36237 } }
I20260923 20:13:52.224457 19401 leader_election.cc:288] T e433fc8d3d0f4107a3da1eb05040f2e1 P dee43928c80f40da85ba7bb7bf49882c [CANDIDATE]: Term 1 election: Requested vote from peers 432510c38c6c4cac9557871bf41b4112 (127.1.170.130:40637), 91fa880bb14b4b329f49778aa7320121 (127.1.170.131:36237)
I20260923 20:13:52.224946 19331 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e433fc8d3d0f4107a3da1eb05040f2e1" candidate_uuid: "dee43928c80f40da85ba7bb7bf49882c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "91fa880bb14b4b329f49778aa7320121"
I20260923 20:13:52.224968 19197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e433fc8d3d0f4107a3da1eb05040f2e1" candidate_uuid: "dee43928c80f40da85ba7bb7bf49882c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "432510c38c6c4cac9557871bf41b4112"
I20260923 20:13:52.225172 19197 raft_consensus.cc:3037] T e433fc8d3d0f4107a3da1eb05040f2e1 P 432510c38c6c4cac9557871bf41b4112 [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:13:52.225154 19331 raft_consensus.cc:3037] T e433fc8d3d0f4107a3da1eb05040f2e1 P 91fa880bb14b4b329f49778aa7320121 [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:13:52.227547 19331 raft_consensus.cc:2454] T e433fc8d3d0f4107a3da1eb05040f2e1 P 91fa880bb14b4b329f49778aa7320121 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dee43928c80f40da85ba7bb7bf49882c in term 1.
I20260923 20:13:52.227612 19197 raft_consensus.cc:2454] T e433fc8d3d0f4107a3da1eb05040f2e1 P 432510c38c6c4cac9557871bf41b4112 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dee43928c80f40da85ba7bb7bf49882c in term 1.
I20260923 20:13:52.228012 18996 leader_election.cc:302] T e433fc8d3d0f4107a3da1eb05040f2e1 P dee43928c80f40da85ba7bb7bf49882c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 91fa880bb14b4b329f49778aa7320121, dee43928c80f40da85ba7bb7bf49882c; no voters:
I20260923 20:13:52.228324 19401 raft_consensus.cc:2781] T e433fc8d3d0f4107a3da1eb05040f2e1 P dee43928c80f40da85ba7bb7bf49882c [term 1 FOLLOWER]: Leader election won for term 1
I20260923 20:13:52.228786 19401 raft_consensus.cc:686] T e433fc8d3d0f4107a3da1eb05040f2e1 P dee43928c80f40da85ba7bb7bf49882c [term 1 LEADER]: Becoming Leader. State: Replica: dee43928c80f40da85ba7bb7bf49882c, State: Running, Role: LEADER
I20260923 20:13:52.229363 19401 consensus_queue.cc:237] T e433fc8d3d0f4107a3da1eb05040f2e1 P dee43928c80f40da85ba7bb7bf49882c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dee43928c80f40da85ba7bb7bf49882c" member_type: VOTER last_known_addr { host: "127.1.170.129" port: 32933 } } peers { permanent_uuid: "432510c38c6c4cac9557871bf41b4112" member_type: VOTER last_known_addr { host: "127.1.170.130" port: 40637 } } peers { permanent_uuid: "91fa880bb14b4b329f49778aa7320121" member_type: VOTER last_known_addr { host: "127.1.170.131" port: 36237 } }
I20260923 20:13:52.234275 18917 catalog_manager.cc:5495] T e433fc8d3d0f4107a3da1eb05040f2e1 P dee43928c80f40da85ba7bb7bf49882c reported cstate change: term changed from 0 to 1, leader changed from <none> to dee43928c80f40da85ba7bb7bf49882c (127.1.170.129). New cstate: current_term: 1 leader_uuid: "dee43928c80f40da85ba7bb7bf49882c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dee43928c80f40da85ba7bb7bf49882c" member_type: VOTER last_known_addr { host: "127.1.170.129" port: 32933 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "432510c38c6c4cac9557871bf41b4112" member_type: VOTER last_known_addr { host: "127.1.170.130" port: 40637 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "91fa880bb14b4b329f49778aa7320121" member_type: VOTER last_known_addr { host: "127.1.170.131" port: 36237 } health_report { overall_health: UNKNOWN } } }
I20260923 20:13:52.310523 19390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:13:52.303603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56856 (local address 127.1.170.129:32933)
0923 20:13:52.303795 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0923 20:13:52.303799 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 20:13:52.304184 (+ 385us) server_negotiation.cc:407] Connection header received
0923 20:13:52.304355 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:13:52.304361 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:13:52.304464 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:13:52.304681 (+ 217us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 20:13:52.305503 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:13:52.306411 (+ 908us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:13:52.308234 (+ 1823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:13:52.308649 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:13:52.308747 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:13:52.309337 (+ 590us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 20:13:52.309981 (+ 644us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 20:13:52.310105 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0923 20:13:52.310231 (+ 126us) server_negotiation.cc:299] Negotiation successful
0923 20:13:52.310303 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260923 20:13:52.319911 19331 raft_consensus.cc:1261] T e433fc8d3d0f4107a3da1eb05040f2e1 P 91fa880bb14b4b329f49778aa7320121 [term 1 FOLLOWER]: Refusing update from remote peer dee43928c80f40da85ba7bb7bf49882c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 20:13:52.319934 19197 raft_consensus.cc:1261] T e433fc8d3d0f4107a3da1eb05040f2e1 P 432510c38c6c4cac9557871bf41b4112 [term 1 FOLLOWER]: Refusing update from remote peer dee43928c80f40da85ba7bb7bf49882c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 20:13:52.321394 19406 consensus_queue.cc:1035] T e433fc8d3d0f4107a3da1eb05040f2e1 P dee43928c80f40da85ba7bb7bf49882c [LEADER]: Connected to new peer: Peer: permanent_uuid: "91fa880bb14b4b329f49778aa7320121" member_type: VOTER last_known_addr { host: "127.1.170.131" port: 36237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 20:13:52.321955 19401 consensus_queue.cc:1035] T e433fc8d3d0f4107a3da1eb05040f2e1 P dee43928c80f40da85ba7bb7bf49882c [LEADER]: Connected to new peer: Peer: permanent_uuid: "432510c38c6c4cac9557871bf41b4112" member_type: VOTER last_known_addr { host: "127.1.170.130" port: 40637 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 20:13:52.336119 19410 mvcc.cc:205] Tried to move back new op lower bound from 7332636394768781312 to 7332636394415579136. Current Snapshot: MvccSnapshot[applied={T|T < 7332636394768781312}]
I20260923 20:13:52.354667 18919 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51372:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 20:13:52.355020 18919 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51372:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 20:13:52.359115 18919 catalog_manager.cc:5782] T 00000000000000000000000000000000 P aa87f9100b0c4ae3932ca34827cf796a: Sending DeleteTablet for 3 replicas of tablet e433fc8d3d0f4107a3da1eb05040f2e1
I20260923 20:13:52.360440 19043 tablet_service.cc:1514] Processing DeleteTablet for tablet e433fc8d3d0f4107a3da1eb05040f2e1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 20:13:52 UTC) from {username='kudu', principal='kudu/127.1.170.190@KRBTEST.COM'} at 127.0.0.1:56840
I20260923 20:13:52.360426 19177 tablet_service.cc:1514] Processing DeleteTablet for tablet e433fc8d3d0f4107a3da1eb05040f2e1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 20:13:52 UTC) from {username='kudu', principal='kudu/127.1.170.190@KRBTEST.COM'} at 127.0.0.1:55550
I20260923 20:13:52.361207 19311 tablet_service.cc:1514] Processing DeleteTablet for tablet e433fc8d3d0f4107a3da1eb05040f2e1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 20:13:52 UTC) from {username='kudu', principal='kudu/127.1.170.190@KRBTEST.COM'} at 127.0.0.1:50620
I20260923 20:13:52.362608 1706 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskqUuku6/build/asan/bin/kudu pid 18980
I20260923 20:13:52.362871 19422 tablet_replica.cc:330] T e433fc8d3d0f4107a3da1eb05040f2e1 P 91fa880bb14b4b329f49778aa7320121: stopping tablet replica
I20260923 20:13:52.363320 19422 raft_consensus.cc:2229] T e433fc8d3d0f4107a3da1eb05040f2e1 P 91fa880bb14b4b329f49778aa7320121 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 20:13:52.363875 19422 raft_consensus.cc:2258] T e433fc8d3d0f4107a3da1eb05040f2e1 P 91fa880bb14b4b329f49778aa7320121 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 20:13:52.366119 19420 tablet_replica.cc:330] T e433fc8d3d0f4107a3da1eb05040f2e1 P dee43928c80f40da85ba7bb7bf49882c: stopping tablet replica
I20260923 20:13:52.366731 19420 raft_consensus.cc:2229] T e433fc8d3d0f4107a3da1eb05040f2e1 P dee43928c80f40da85ba7bb7bf49882c [term 1 LEADER]: Raft consensus shutting down.
I20260923 20:13:52.367619 19420 raft_consensus.cc:2258] T e433fc8d3d0f4107a3da1eb05040f2e1 P dee43928c80f40da85ba7bb7bf49882c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 20:13:52.367693 19422 ts_tablet_manager.cc:1905] T e433fc8d3d0f4107a3da1eb05040f2e1 P 91fa880bb14b4b329f49778aa7320121: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 20:13:52.368209 19421 tablet_replica.cc:330] T e433fc8d3d0f4107a3da1eb05040f2e1 P 432510c38c6c4cac9557871bf41b4112: stopping tablet replica
I20260923 20:13:52.368778 19421 raft_consensus.cc:2229] T e433fc8d3d0f4107a3da1eb05040f2e1 P 432510c38c6c4cac9557871bf41b4112 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 20:13:52.369448 19421 raft_consensus.cc:2258] T e433fc8d3d0f4107a3da1eb05040f2e1 P 432510c38c6c4cac9557871bf41b4112 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 20:13:52.370478 19420 ts_tablet_manager.cc:1905] T e433fc8d3d0f4107a3da1eb05040f2e1 P dee43928c80f40da85ba7bb7bf49882c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 20:13:52.373011 19421 ts_tablet_manager.cc:1905] T e433fc8d3d0f4107a3da1eb05040f2e1 P 432510c38c6c4cac9557871bf41b4112: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 20:13:52.378525 19420 ts_tablet_manager.cc:1918] T e433fc8d3d0f4107a3da1eb05040f2e1 P dee43928c80f40da85ba7bb7bf49882c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 20:13:52.378728 19420 log.cc:1196] T e433fc8d3d0f4107a3da1eb05040f2e1 P dee43928c80f40da85ba7bb7bf49882c: Deleting WAL directory at /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790194121577027-1706-0/minicluster-data/ts-0/wal/wals/e433fc8d3d0f4107a3da1eb05040f2e1
I20260923 20:13:52.379361 19420 ts_tablet_manager.cc:1939] T e433fc8d3d0f4107a3da1eb05040f2e1 P dee43928c80f40da85ba7bb7bf49882c: Deleting consensus metadata
I20260923 20:13:52.381508 18905 catalog_manager.cc:4841] TS dee43928c80f40da85ba7bb7bf49882c (127.1.170.129:32933): tablet e433fc8d3d0f4107a3da1eb05040f2e1 (table test-table [id=edd9079660fb4828b9f74527c18e2790]) successfully deleted
I20260923 20:13:52.387094 19421 ts_tablet_manager.cc:1918] T e433fc8d3d0f4107a3da1eb05040f2e1 P 432510c38c6c4cac9557871bf41b4112: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 20:13:52.387303 19421 log.cc:1196] T e433fc8d3d0f4107a3da1eb05040f2e1 P 432510c38c6c4cac9557871bf41b4112: Deleting WAL directory at /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790194121577027-1706-0/minicluster-data/ts-1/wal/wals/e433fc8d3d0f4107a3da1eb05040f2e1
I20260923 20:13:52.387928 19421 ts_tablet_manager.cc:1939] T e433fc8d3d0f4107a3da1eb05040f2e1 P 432510c38c6c4cac9557871bf41b4112: Deleting consensus metadata
I20260923 20:13:52.388622 19422 ts_tablet_manager.cc:1918] T e433fc8d3d0f4107a3da1eb05040f2e1 P 91fa880bb14b4b329f49778aa7320121: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 20:13:52.388868 19422 log.cc:1196] T e433fc8d3d0f4107a3da1eb05040f2e1 P 91fa880bb14b4b329f49778aa7320121: Deleting WAL directory at /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790194121577027-1706-0/minicluster-data/ts-2/wal/wals/e433fc8d3d0f4107a3da1eb05040f2e1
I20260923 20:13:52.389454 19422 ts_tablet_manager.cc:1939] T e433fc8d3d0f4107a3da1eb05040f2e1 P 91fa880bb14b4b329f49778aa7320121: Deleting consensus metadata
I20260923 20:13:52.390074 18904 catalog_manager.cc:4841] TS 432510c38c6c4cac9557871bf41b4112 (127.1.170.130:40637): tablet e433fc8d3d0f4107a3da1eb05040f2e1 (table test-table [id=edd9079660fb4828b9f74527c18e2790]) successfully deleted
Sep 23 20:13:52 dist-test-slave-th4g krb5kdc[18874](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194429, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.170.129@KRBTEST.COM
I20260923 20:13:52.391366 18903 catalog_manager.cc:4841] TS 91fa880bb14b4b329f49778aa7320121 (127.1.170.131:36237): tablet e433fc8d3d0f4107a3da1eb05040f2e1 (table test-table [id=edd9079660fb4828b9f74527c18e2790]) successfully deleted
I20260923 20:13:52.399883 19390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:13:52.364093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56862 (local address 127.1.170.129:32933)
0923 20:13:52.364292 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0923 20:13:52.364300 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 20:13:52.379280 (+ 14980us) server_negotiation.cc:407] Connection header received
0923 20:13:52.379659 (+ 379us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:13:52.379665 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:13:52.379770 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:13:52.379887 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:13:52.381993 (+ 2106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:13:52.383278 (+ 1285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:13:52.385337 (+ 2059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:13:52.386090 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:13:52.386223 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:13:52.392060 (+ 5837us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:13:52.392087 (+ 27us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:13:52.392115 (+ 28us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:13:52.392178 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:13:52.396165 (+ 3987us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:13:52.397432 (+ 1267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:13:52.397445 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:13:52.397452 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:13:52.397651 (+ 199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:13:52.398219 (+ 568us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:13:52.398223 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:13:52.398225 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:13:52.398648 (+ 423us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:13:52.398903 (+ 255us) server_negotiation.cc:1035] Waiting for connection context
0923 20:13:52.399350 (+ 447us) server_negotiation.cc:299] Negotiation successful
0923 20:13:52.399550 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260923 20:13:52.400635 19103 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:13:52.505055 1706 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqUuku6/build/asan/bin/kudu with pid 18980
I20260923 20:13:52.524802 1706 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskqUuku6/build/asan/bin/kudu pid 19114
Sep 23 20:13:52 dist-test-slave-th4g krb5kdc[18874](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194429, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.170.130@KRBTEST.COM
I20260923 20:13:52.541196 19392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:13:52.526065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55560 (local address 127.1.170.130:40637)
0923 20:13:52.526244 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0923 20:13:52.526249 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 20:13:52.526269 (+ 20us) server_negotiation.cc:407] Connection header received
0923 20:13:52.526430 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:13:52.526434 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:13:52.526498 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:13:52.526590 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:13:52.527974 (+ 1384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:13:52.528711 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:13:52.530148 (+ 1437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:13:52.530587 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:13:52.530686 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:13:52.535342 (+ 4656us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:13:52.535359 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:13:52.535394 (+ 35us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:13:52.535440 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:13:52.538557 (+ 3117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:13:52.539345 (+ 788us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:13:52.539353 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:13:52.539356 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:13:52.539498 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:13:52.539862 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:13:52.539866 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:13:52.539868 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:13:52.540254 (+ 386us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:13:52.540495 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0923 20:13:52.540765 (+ 270us) server_negotiation.cc:299] Negotiation successful
0923 20:13:52.540943 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260923 20:13:52.542433 19237 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:13:52.633203 1706 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqUuku6/build/asan/bin/kudu with pid 19114
I20260923 20:13:52.650465 1706 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskqUuku6/build/asan/bin/kudu pid 19248
Sep 23 20:13:52 dist-test-slave-th4g krb5kdc[18874](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194429, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.170.131@KRBTEST.COM
I20260923 20:13:52.667482 19394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:13:52.651327 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50630 (local address 127.1.170.131:36237)
0923 20:13:52.651501 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0923 20:13:52.651505 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 20:13:52.651523 (+ 18us) server_negotiation.cc:407] Connection header received
0923 20:13:52.651606 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:13:52.651610 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:13:52.651672 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:13:52.651751 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:13:52.653118 (+ 1367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:13:52.653862 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:13:52.655367 (+ 1505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:13:52.655828 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:13:52.655947 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:13:52.660634 (+ 4687us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:13:52.660672 (+ 38us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:13:52.660690 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:13:52.660730 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:13:52.664101 (+ 3371us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:13:52.664875 (+ 774us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:13:52.664882 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:13:52.664885 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:13:52.665048 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:13:52.665372 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:13:52.665375 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:13:52.665377 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:13:52.665731 (+ 354us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:13:52.666153 (+ 422us) server_negotiation.cc:1035] Waiting for connection context
0923 20:13:52.666651 (+ 498us) server_negotiation.cc:299] Negotiation successful
0923 20:13:52.667043 (+ 392us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260923 20:13:52.668224 19371 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:13:52.758248 1706 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqUuku6/build/asan/bin/kudu with pid 19248
I20260923 20:13:52.775449 1706 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskqUuku6/build/asan/bin/kudu pid 18887
I20260923 20:13:52.776576 18949 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:13:52.889482 1706 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqUuku6/build/asan/bin/kudu with pid 18887
2026-09-23T20:13:52Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (4694 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-09-23T20:13:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T20:13:52Z Disabled control of system clock
I20260923 20:13:52.971181 1706 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqUuku6/build/asan/bin/kudu
/tmp/dist-test-taskqUuku6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790194121577027-1706-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790194121577027-1706-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790194121577027-1706-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790194121577027-1706-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.170.190:42127
--webserver_interface=127.1.170.190
--webserver_port=0
--builtin_ntp_servers=127.1.170.148:39963
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.170.190:42127
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260923 20:13:53.298388 19431 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:13:53.298672 19431 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:13:53.298753 19431 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:13:53.306005 19431 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 20:13:53.306110 19431 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:13:53.306147 19431 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:13:53.306195 19431 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 20:13:53.306232 19431 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 20:13:53.315816 19431 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.170.148:39963
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790194121577027-1706-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790194121577027-1706-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.170.190:42127
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.170.190:42127
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790194121577027-1706-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.170.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790194121577027-1706-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:13:53.317101 19431 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:13:53.318910 19431 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:13:53.330843 19439 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:13:53.330984 19437 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:13:53.331015 19436 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:13:53.331593 19431 server_base.cc:1034] running on GCE node
I20260923 20:13:53.332362 19431 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:13:53.333766 19431 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:13:53.335009 19431 hybrid_clock.cc:648] HybridClock initialized: now 1790194433334971 us; error 69 us; skew 500 ppm
I20260923 20:13:53.335373 19431 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260923 20:13:53.337697 19431 webserver.cc:460] Webserver started at http://127.1.170.190:34825/ using document root <none> and password file <none>
I20260923 20:13:53.338438 19431 fs_manager.cc:359] Metadata directory not provided
I20260923 20:13:53.338562 19431 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:13:53.338886 19431 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:13:53.341769 19431 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790194121577027-1706-0/minicluster-data/master-0/data/instance:
uuid: "380fc491ed614a1782495140d75136d0"
format_stamp: "Formatted at 2026-09-23 20:13:53 on dist-test-slave-th4g"
server_key: "81db3247e2b2177bb404e878e97acb7b"
server_key_iv: "ad6bf1eeba2074f1f2d766e4aeb0fdb3"
server_key_version: "encryptionkey@0"
I20260923 20:13:53.342586 19431 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790194121577027-1706-0/minicluster-data/master-0/wal/instance:
uuid: "380fc491ed614a1782495140d75136d0"
format_stamp: "Formatted at 2026-09-23 20:13:53 on dist-test-slave-th4g"
server_key: "81db3247e2b2177bb404e878e97acb7b"
server_key_iv: "ad6bf1eeba2074f1f2d766e4aeb0fdb3"
server_key_version: "encryptionkey@0"
I20260923 20:13:53.348263 19431 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260923 20:13:53.351629 19445 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:13:53.352993 19431 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260923 20:13:53.353180 19431 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790194121577027-1706-0/minicluster-data/master-0/data,/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790194121577027-1706-0/minicluster-data/master-0/wal
uuid: "380fc491ed614a1782495140d75136d0"
format_stamp: "Formatted at 2026-09-23 20:13:53 on dist-test-slave-th4g"
server_key: "81db3247e2b2177bb404e878e97acb7b"
server_key_iv: "ad6bf1eeba2074f1f2d766e4aeb0fdb3"
server_key_version: "encryptionkey@0"
I20260923 20:13:53.353394 19431 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790194121577027-1706-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790194121577027-1706-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790194121577027-1706-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:13:53.366389 19431 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:13:53.367154 19431 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:13:53.367430 19431 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:13:53.385562 19431 rpc_server.cc:307] RPC server started. Bound to: 127.1.170.190:42127
I20260923 20:13:53.385594 19496 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.170.190:42127 every 8 connection(s)
I20260923 20:13:53.386896 19431 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790194121577027-1706-0/minicluster-data/master-0/data/info.pb
I20260923 20:13:53.390482 19497 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:13:53.393396 1706 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqUuku6/build/asan/bin/kudu as pid 19431
I20260923 20:13:53.393579 1706 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790194121577027-1706-0/minicluster-data/master-0/wal/instance
I20260923 20:13:53.394187 1706 external_mini_cluster.cc:1428] Setting key abf1186dc8983d519e2ec252c350e151
I20260923 20:13:53.402441 19497 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 380fc491ed614a1782495140d75136d0: Bootstrap starting.
I20260923 20:13:53.405733 19497 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 380fc491ed614a1782495140d75136d0: Neither blocks nor log segments found. Creating new log.
I20260923 20:13:53.406934 19497 log.cc:824] T 00000000000000000000000000000000 P 380fc491ed614a1782495140d75136d0: Log is configured to *not* fsync() on all Append() calls
I20260923 20:13:53.407812 19500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:13:53.396286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52098 (local address 127.1.170.190:42127)
0923 20:13:53.397333 (+ 1047us) server_negotiation.cc:206] Beginning negotiation
0923 20:13:53.397345 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0923 20:13:53.397386 (+ 41us) server_negotiation.cc:407] Connection header received
0923 20:13:53.399110 (+ 1724us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:13:53.399131 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:13:53.399586 (+ 455us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:13:53.399982 (+ 396us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:13:53.400546 (+ 564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:13:53.401786 (+ 1240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:13:53.403248 (+ 1462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:13:53.403944 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:13:53.404143 (+ 199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:13:53.405316 (+ 1173us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:13:53.405323 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:13:53.405341 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 20:13:53.405942 (+ 601us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:13:53.406049 (+ 107us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 20:13:53.406071 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:13:53.406197 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0923 20:13:53.406420 (+ 223us) server_negotiation.cc:299] Negotiation successful
0923 20:13:53.406706 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":632,"thread_start_us":360,"threads_started":1}
I20260923 20:13:53.410040 19497 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 380fc491ed614a1782495140d75136d0: No bootstrap required, opened a new log
I20260923 20:13:53.417433 19497 raft_consensus.cc:348] T 00000000000000000000000000000000 P 380fc491ed614a1782495140d75136d0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "380fc491ed614a1782495140d75136d0" member_type: VOTER last_known_addr { host: "127.1.170.190" port: 42127 } }
I20260923 20:13:53.417807 19497 raft_consensus.cc:374] T 00000000000000000000000000000000 P 380fc491ed614a1782495140d75136d0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:13:53.417925 19497 raft_consensus.cc:729] T 00000000000000000000000000000000 P 380fc491ed614a1782495140d75136d0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 380fc491ed614a1782495140d75136d0, State: Initialized, Role: FOLLOWER
I20260923 20:13:53.418587 19497 consensus_queue.cc:260] T 00000000000000000000000000000000 P 380fc491ed614a1782495140d75136d0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "380fc491ed614a1782495140d75136d0" member_type: VOTER last_known_addr { host: "127.1.170.190" port: 42127 } }
I20260923 20:13:53.418823 19497 raft_consensus.cc:388] T 00000000000000000000000000000000 P 380fc491ed614a1782495140d75136d0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 20:13:53.418962 19497 raft_consensus.cc:482] T 00000000000000000000000000000000 P 380fc491ed614a1782495140d75136d0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 20:13:53.419137 19497 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 380fc491ed614a1782495140d75136d0 [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:13:53.421476 19497 raft_consensus.cc:504] T 00000000000000000000000000000000 P 380fc491ed614a1782495140d75136d0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "380fc491ed614a1782495140d75136d0" member_type: VOTER last_known_addr { host: "127.1.170.190" port: 42127 } }
I20260923 20:13:53.422042 19497 leader_election.cc:302] T 00000000000000000000000000000000 P 380fc491ed614a1782495140d75136d0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 380fc491ed614a1782495140d75136d0; no voters:
I20260923 20:13:53.422667 19497 leader_election.cc:288] T 00000000000000000000000000000000 P 380fc491ed614a1782495140d75136d0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 20:13:53.422888 19502 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 380fc491ed614a1782495140d75136d0 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 20:13:53.423588 19502 raft_consensus.cc:686] T 00000000000000000000000000000000 P 380fc491ed614a1782495140d75136d0 [term 1 LEADER]: Becoming Leader. State: Replica: 380fc491ed614a1782495140d75136d0, State: Running, Role: LEADER
I20260923 20:13:53.424055 19502 consensus_queue.cc:237] T 00000000000000000000000000000000 P 380fc491ed614a1782495140d75136d0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "380fc491ed614a1782495140d75136d0" member_type: VOTER last_known_addr { host: "127.1.170.190" port: 42127 } }
I20260923 20:13:53.424610 19497 sys_catalog.cc:564] T 00000000000000000000000000000000 P 380fc491ed614a1782495140d75136d0 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 20:13:53.426682 19503 sys_catalog.cc:455] T 00000000000000000000000000000000 P 380fc491ed614a1782495140d75136d0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "380fc491ed614a1782495140d75136d0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "380fc491ed614a1782495140d75136d0" member_type: VOTER last_known_addr { host: "127.1.170.190" port: 42127 } } }
I20260923 20:13:53.426781 19504 sys_catalog.cc:455] T 00000000000000000000000000000000 P 380fc491ed614a1782495140d75136d0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 380fc491ed614a1782495140d75136d0. Latest consensus state: current_term: 1 leader_uuid: "380fc491ed614a1782495140d75136d0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "380fc491ed614a1782495140d75136d0" member_type: VOTER last_known_addr { host: "127.1.170.190" port: 42127 } } }
I20260923 20:13:53.427559 19503 sys_catalog.cc:458] T 00000000000000000000000000000000 P 380fc491ed614a1782495140d75136d0 [sys.catalog]: This master's current role is: LEADER
I20260923 20:13:53.427795 19504 sys_catalog.cc:458] T 00000000000000000000000000000000 P 380fc491ed614a1782495140d75136d0 [sys.catalog]: This master's current role is: LEADER
I20260923 20:13:53.428819 19510 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 20:13:53.433928 19510 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 20:13:53.444059 19510 catalog_manager.cc:1347] Generated new cluster ID: 335c9c39174f4bcf9994d010a9aec801
I20260923 20:13:53.444167 19510 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 20:13:53.455958 19510 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 20:13:53.457342 19510 catalog_manager.cc:1504] Loading token signing keys...
I20260923 20:13:53.465030 19510 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 380fc491ed614a1782495140d75136d0: Generated new TSK 0
I20260923 20:13:53.465786 19510 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 20:13:53.475894 1706 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqUuku6/build/asan/bin/kudu
/tmp/dist-test-taskqUuku6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790194121577027-1706-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790194121577027-1706-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790194121577027-1706-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790194121577027-1706-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.170.129:0
--local_ip_for_outbound_sockets=127.1.170.129
--webserver_interface=127.1.170.129
--webserver_port=0
--tserver_master_addrs=127.1.170.190:42127
--builtin_ntp_servers=127.1.170.148:39963
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260923 20:13:53.808511 19521 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:13:53.808782 19521 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:13:53.808925 19521 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:13:53.816120 19521 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:13:53.816215 19521 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:13:53.816399 19521 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.170.129
I20260923 20:13:53.825531 19521 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.170.148:39963
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790194121577027-1706-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790194121577027-1706-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.170.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790194121577027-1706-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.170.129
--webserver_port=0
--tserver_master_addrs=127.1.170.190:42127
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.170.129
--log_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790194121577027-1706-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:13:53.826802 19521 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:13:53.828493 19521 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:13:53.841771 19526 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:13:53.841732 19529 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:13:53.841815 19527 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:13:53.847357 19521 server_base.cc:1034] running on GCE node
I20260923 20:13:53.847992 19521 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:13:53.849015 19521 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:13:53.850267 19521 hybrid_clock.cc:648] HybridClock initialized: now 1790194433850221 us; error 50 us; skew 500 ppm
I20260923 20:13:53.850639 19521 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260923 20:13:53.852846 19521 webserver.cc:460] Webserver started at http://127.1.170.129:42909/ using document root <none> and password file <none>
I20260923 20:13:53.853528 19521 fs_manager.cc:359] Metadata directory not provided
I20260923 20:13:53.853623 19521 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:13:53.854001 19521 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:13:53.856796 19521 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790194121577027-1706-0/minicluster-data/ts-0/data/instance:
uuid: "212456b209c24be897b2edbfd782bdc4"
format_stamp: "Formatted at 2026-09-23 20:13:53 on dist-test-slave-th4g"
server_key: "cf52b8c8d92b06d7a1b23e5667624f80"
server_key_iv: "a34383403db64b30eae78e43e0554b6c"
server_key_version: "encryptionkey@0"
I20260923 20:13:53.857527 19521 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790194121577027-1706-0/minicluster-data/ts-0/wal/instance:
uuid: "212456b209c24be897b2edbfd782bdc4"
format_stamp: "Formatted at 2026-09-23 20:13:53 on dist-test-slave-th4g"
server_key: "cf52b8c8d92b06d7a1b23e5667624f80"
server_key_iv: "a34383403db64b30eae78e43e0554b6c"
server_key_version: "encryptionkey@0"
I20260923 20:13:53.862668 19521 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.001s
I20260923 20:13:53.866089 19535 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:13:53.867324 19521 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260923 20:13:53.867502 19521 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790194121577027-1706-0/minicluster-data/ts-0/data,/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790194121577027-1706-0/minicluster-data/ts-0/wal
uuid: "212456b209c24be897b2edbfd782bdc4"
format_stamp: "Formatted at 2026-09-23 20:13:53 on dist-test-slave-th4g"
server_key: "cf52b8c8d92b06d7a1b23e5667624f80"
server_key_iv: "a34383403db64b30eae78e43e0554b6c"
server_key_version: "encryptionkey@0"
I20260923 20:13:53.867718 19521 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790194121577027-1706-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790194121577027-1706-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790194121577027-1706-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:13:53.883756 19521 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:13:53.884557 19521 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:13:53.884847 19521 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:13:53.885834 19521 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 20:13:53.887292 19521 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 20:13:53.887406 19521 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 20:13:53.887534 19521 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 20:13:53.887626 19521 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 20:13:53.920955 19521 rpc_server.cc:307] RPC server started. Bound to: 127.1.170.129:43077
I20260923 20:13:53.921043 19647 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.170.129:43077 every 8 connection(s)
I20260923 20:13:53.922353 19521 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790194121577027-1706-0/minicluster-data/ts-0/data/info.pb
I20260923 20:13:53.932008 1706 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqUuku6/build/asan/bin/kudu as pid 19521
I20260923 20:13:53.932201 1706 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790194121577027-1706-0/minicluster-data/ts-0/wal/instance
I20260923 20:13:53.932679 1706 external_mini_cluster.cc:1428] Setting key e57892e2f3012cfd8b98147c4d4865aa
I20260923 20:13:53.935907 1706 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqUuku6/build/asan/bin/kudu
/tmp/dist-test-taskqUuku6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790194121577027-1706-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790194121577027-1706-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790194121577027-1706-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790194121577027-1706-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.170.130:0
--local_ip_for_outbound_sockets=127.1.170.130
--webserver_interface=127.1.170.130
--webserver_port=0
--tserver_master_addrs=127.1.170.190:42127
--builtin_ntp_servers=127.1.170.148:39963
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260923 20:13:53.936082 19650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:13:53.924961 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.170.129:60913 (local address 127.1.170.190:42127)
0923 20:13:53.925434 (+ 473us) server_negotiation.cc:206] Beginning negotiation
0923 20:13:53.925440 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:13:53.926768 (+ 1328us) server_negotiation.cc:407] Connection header received
0923 20:13:53.928634 (+ 1866us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:13:53.928639 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:13:53.928716 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:13:53.928816 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:13:53.930112 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:13:53.930882 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:13:53.932674 (+ 1792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:13:53.933265 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:13:53.933384 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:13:53.935057 (+ 1673us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:13:53.935065 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:13:53.935070 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 20:13:53.935091 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:13:53.935150 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 20:13:53.935167 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:13:53.935287 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0923 20:13:53.935558 (+ 271us) server_negotiation.cc:299] Negotiation successful
0923 20:13:53.935675 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":337,"thread_start_us":222,"threads_started":1}
I20260923 20:13:53.937064 19651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:13:53.925636 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.190:42127 (local address 127.1.170.129:60913)
0923 20:13:53.926613 (+ 977us) negotiation.cc:107] Waiting for socket to connect
0923 20:13:53.926652 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0923 20:13:53.928323 (+ 1671us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:13:53.929110 (+ 787us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:13:53.929123 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:13:53.929216 (+ 93us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:13:53.929939 (+ 723us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:13:53.929959 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:13:53.931027 (+ 1068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:13:53.931030 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:13:53.932528 (+ 1498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:13:53.932541 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:13:53.933563 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:13:53.933578 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:13:53.933823 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:13:53.934674 (+ 851us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 20:13:53.934677 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:13:53.934724 (+ 47us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 20:13:53.934861 (+ 137us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:13:53.935298 (+ 437us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:13:53.935307 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:13:53.935315 (+ 8us) client_negotiation.cc:769] Sending connection context
0923 20:13:53.935488 (+ 173us) client_negotiation.cc:240] Negotiation successful
0923 20:13:53.935828 (+ 340us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":522,"thread_start_us":321,"threads_started":1}
I20260923 20:13:53.938860 19648 heartbeater.cc:347] Connected to a master server at 127.1.170.190:42127
I20260923 20:13:53.939204 19648 heartbeater.cc:464] Registering TS with master...
I20260923 20:13:53.940172 19648 heartbeater.cc:511] Master 127.1.170.190:42127 requested a full tablet report, sending...
I20260923 20:13:53.942396 19462 ts_manager.cc:194] Registered new tserver with Master: 212456b209c24be897b2edbfd782bdc4 (127.1.170.129:43077)
I20260923 20:13:53.943846 19462 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.170.129:60913
W20260923 20:13:54.263131 19652 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:13:54.263423 19652 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:13:54.263509 19652 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:13:54.271077 19652 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:13:54.271199 19652 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:13:54.271311 19652 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.170.130
I20260923 20:13:54.281641 19652 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.170.148:39963
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790194121577027-1706-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790194121577027-1706-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.170.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790194121577027-1706-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.170.130
--webserver_port=0
--tserver_master_addrs=127.1.170.190:42127
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.170.130
--log_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790194121577027-1706-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:13:54.283027 19652 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:13:54.284809 19652 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:13:54.297897 19660 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:13:54.297951 19658 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:13:54.297974 19657 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:13:54.298576 19652 server_base.cc:1034] running on GCE node
I20260923 20:13:54.299189 19652 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:13:54.300165 19652 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:13:54.301407 19652 hybrid_clock.cc:648] HybridClock initialized: now 1790194434301345 us; error 50 us; skew 500 ppm
I20260923 20:13:54.301746 19652 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260923 20:13:54.304124 19652 webserver.cc:460] Webserver started at http://127.1.170.130:39293/ using document root <none> and password file <none>
I20260923 20:13:54.304847 19652 fs_manager.cc:359] Metadata directory not provided
I20260923 20:13:54.305011 19652 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:13:54.305336 19652 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:13:54.308220 19652 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790194121577027-1706-0/minicluster-data/ts-1/data/instance:
uuid: "fbcc86fecac3454fbd3568c27f3d26a6"
format_stamp: "Formatted at 2026-09-23 20:13:54 on dist-test-slave-th4g"
server_key: "c5e8d08f39d86163c91dcd350830d629"
server_key_iv: "b49252be587e7d567a826573215e06d8"
server_key_version: "encryptionkey@0"
I20260923 20:13:54.309058 19652 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790194121577027-1706-0/minicluster-data/ts-1/wal/instance:
uuid: "fbcc86fecac3454fbd3568c27f3d26a6"
format_stamp: "Formatted at 2026-09-23 20:13:54 on dist-test-slave-th4g"
server_key: "c5e8d08f39d86163c91dcd350830d629"
server_key_iv: "b49252be587e7d567a826573215e06d8"
server_key_version: "encryptionkey@0"
I20260923 20:13:54.314203 19652 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260923 20:13:54.317490 19666 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:13:54.318776 19652 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 20:13:54.318945 19652 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790194121577027-1706-0/minicluster-data/ts-1/data,/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790194121577027-1706-0/minicluster-data/ts-1/wal
uuid: "fbcc86fecac3454fbd3568c27f3d26a6"
format_stamp: "Formatted at 2026-09-23 20:13:54 on dist-test-slave-th4g"
server_key: "c5e8d08f39d86163c91dcd350830d629"
server_key_iv: "b49252be587e7d567a826573215e06d8"
server_key_version: "encryptionkey@0"
I20260923 20:13:54.319165 19652 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790194121577027-1706-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790194121577027-1706-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790194121577027-1706-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:13:54.358371 19652 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:13:54.359215 19652 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:13:54.359511 19652 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:13:54.360472 19652 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 20:13:54.361969 19652 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 20:13:54.362078 19652 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 20:13:54.362200 19652 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 20:13:54.362288 19652 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 20:13:54.395802 19652 rpc_server.cc:307] RPC server started. Bound to: 127.1.170.130:38621
I20260923 20:13:54.395888 19778 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.170.130:38621 every 8 connection(s)
I20260923 20:13:54.397042 19652 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790194121577027-1706-0/minicluster-data/ts-1/data/info.pb
I20260923 20:13:54.402227 1706 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqUuku6/build/asan/bin/kudu as pid 19652
I20260923 20:13:54.402417 1706 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790194121577027-1706-0/minicluster-data/ts-1/wal/instance
I20260923 20:13:54.402858 1706 external_mini_cluster.cc:1428] Setting key efc2faa513f24b49e337e71f221afc03
I20260923 20:13:54.405841 1706 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqUuku6/build/asan/bin/kudu
/tmp/dist-test-taskqUuku6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790194121577027-1706-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790194121577027-1706-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790194121577027-1706-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790194121577027-1706-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.170.131:0
--local_ip_for_outbound_sockets=127.1.170.131
--webserver_interface=127.1.170.131
--webserver_port=0
--tserver_master_addrs=127.1.170.190:42127
--builtin_ntp_servers=127.1.170.148:39963
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260923 20:13:54.411325 19650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:13:54.399595 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.170.130:49653 (local address 127.1.170.190:42127)
0923 20:13:54.399813 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0923 20:13:54.399819 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:13:54.401363 (+ 1544us) server_negotiation.cc:407] Connection header received
0923 20:13:54.403925 (+ 2562us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:13:54.403930 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:13:54.404009 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:13:54.404124 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:13:54.405484 (+ 1360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:13:54.406619 (+ 1135us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:13:54.408173 (+ 1554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:13:54.408864 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:13:54.408999 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:13:54.410375 (+ 1376us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:13:54.410383 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:13:54.410391 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 20:13:54.410417 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:13:54.410461 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 20:13:54.410480 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:13:54.410573 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0923 20:13:54.410800 (+ 227us) server_negotiation.cc:299] Negotiation successful
0923 20:13:54.410918 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260923 20:13:54.412303 19781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:13:54.400135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.190:42127 (local address 127.1.170.130:49653)
0923 20:13:54.401193 (+ 1058us) negotiation.cc:107] Waiting for socket to connect
0923 20:13:54.401239 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0923 20:13:54.403624 (+ 2385us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:13:54.404368 (+ 744us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:13:54.404379 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:13:54.404489 (+ 110us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:13:54.405259 (+ 770us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:13:54.405287 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:13:54.406784 (+ 1497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:13:54.406788 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:13:54.408026 (+ 1238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:13:54.408039 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:13:54.409073 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:13:54.409077 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:13:54.409270 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:13:54.410051 (+ 781us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 20:13:54.410054 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:13:54.410094 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 20:13:54.410178 (+ 84us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:13:54.410590 (+ 412us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:13:54.410598 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:13:54.410603 (+ 5us) client_negotiation.cc:769] Sending connection context
0923 20:13:54.410753 (+ 150us) client_negotiation.cc:240] Negotiation successful
0923 20:13:54.411056 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":550,"thread_start_us":316,"threads_started":1}
I20260923 20:13:54.413715 19779 heartbeater.cc:347] Connected to a master server at 127.1.170.190:42127
I20260923 20:13:54.414099 19779 heartbeater.cc:464] Registering TS with master...
I20260923 20:13:54.414960 19779 heartbeater.cc:511] Master 127.1.170.190:42127 requested a full tablet report, sending...
I20260923 20:13:54.416437 19462 ts_manager.cc:194] Registered new tserver with Master: fbcc86fecac3454fbd3568c27f3d26a6 (127.1.170.130:38621)
I20260923 20:13:54.417259 19462 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.170.130:49653
W20260923 20:13:54.732184 19782 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:13:54.732439 19782 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:13:54.732602 19782 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:13:54.739790 19782 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:13:54.739884 19782 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:13:54.740064 19782 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.170.131
I20260923 20:13:54.748944 19782 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.170.148:39963
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790194121577027-1706-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790194121577027-1706-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.170.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790194121577027-1706-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.170.131
--webserver_port=0
--tserver_master_addrs=127.1.170.190:42127
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.170.131
--log_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790194121577027-1706-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:13:54.750211 19782 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:13:54.751857 19782 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:13:54.766215 19787 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:13:54.766115 19790 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:13:54.766166 19788 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:13:54.767251 19782 server_base.cc:1034] running on GCE node
I20260923 20:13:54.767819 19782 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:13:54.768703 19782 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:13:54.769960 19782 hybrid_clock.cc:648] HybridClock initialized: now 1790194434769921 us; error 51 us; skew 500 ppm
I20260923 20:13:54.770257 19782 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260923 20:13:54.772549 19782 webserver.cc:460] Webserver started at http://127.1.170.131:35503/ using document root <none> and password file <none>
I20260923 20:13:54.773178 19782 fs_manager.cc:359] Metadata directory not provided
I20260923 20:13:54.773259 19782 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:13:54.773530 19782 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:13:54.776283 19782 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790194121577027-1706-0/minicluster-data/ts-2/data/instance:
uuid: "1c2285e6a7914d55bb93642018d06852"
format_stamp: "Formatted at 2026-09-23 20:13:54 on dist-test-slave-th4g"
server_key: "5b75844ca0c9642db6456fe1e096458c"
server_key_iv: "c4ec065ac04251c4aedb582e691f24b9"
server_key_version: "encryptionkey@0"
I20260923 20:13:54.776986 19782 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790194121577027-1706-0/minicluster-data/ts-2/wal/instance:
uuid: "1c2285e6a7914d55bb93642018d06852"
format_stamp: "Formatted at 2026-09-23 20:13:54 on dist-test-slave-th4g"
server_key: "5b75844ca0c9642db6456fe1e096458c"
server_key_iv: "c4ec065ac04251c4aedb582e691f24b9"
server_key_version: "encryptionkey@0"
I20260923 20:13:54.781792 19782 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260923 20:13:54.785020 19796 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:13:54.786407 19782 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 20:13:54.786594 19782 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790194121577027-1706-0/minicluster-data/ts-2/data,/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790194121577027-1706-0/minicluster-data/ts-2/wal
uuid: "1c2285e6a7914d55bb93642018d06852"
format_stamp: "Formatted at 2026-09-23 20:13:54 on dist-test-slave-th4g"
server_key: "5b75844ca0c9642db6456fe1e096458c"
server_key_iv: "c4ec065ac04251c4aedb582e691f24b9"
server_key_version: "encryptionkey@0"
I20260923 20:13:54.786813 19782 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790194121577027-1706-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790194121577027-1706-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790194121577027-1706-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:13:54.814494 19782 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:13:54.815361 19782 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:13:54.815676 19782 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:13:54.816680 19782 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 20:13:54.818327 19782 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 20:13:54.818418 19782 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 20:13:54.818503 19782 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 20:13:54.818552 19782 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 20:13:54.852242 19782 rpc_server.cc:307] RPC server started. Bound to: 127.1.170.131:46035
I20260923 20:13:54.852331 19908 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.170.131:46035 every 8 connection(s)
I20260923 20:13:54.853541 19782 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790194121577027-1706-0/minicluster-data/ts-2/data/info.pb
I20260923 20:13:54.862617 1706 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqUuku6/build/asan/bin/kudu as pid 19782
I20260923 20:13:54.862792 1706 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790194121577027-1706-0/minicluster-data/ts-2/wal/instance
I20260923 20:13:54.863333 1706 external_mini_cluster.cc:1428] Setting key 715fae668ae34e079c6f45cbcabc6fa6
I20260923 20:13:54.866207 19650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:13:54.856223 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.170.131:55843 (local address 127.1.170.190:42127)
0923 20:13:54.856422 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0923 20:13:54.856426 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 20:13:54.857755 (+ 1329us) server_negotiation.cc:407] Connection header received
0923 20:13:54.859581 (+ 1826us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:13:54.859585 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:13:54.859651 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:13:54.859731 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:13:54.860667 (+ 936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:13:54.861399 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:13:54.863373 (+ 1974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:13:54.863806 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:13:54.863912 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:13:54.865349 (+ 1437us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:13:54.865355 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:13:54.865361 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 20:13:54.865385 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:13:54.865425 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 20:13:54.865440 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:13:54.865536 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0923 20:13:54.865721 (+ 185us) server_negotiation.cc:299] Negotiation successful
0923 20:13:54.865813 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260923 20:13:54.867129 19911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:13:54.856745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.190:42127 (local address 127.1.170.131:55843)
0923 20:13:54.857621 (+ 876us) negotiation.cc:107] Waiting for socket to connect
0923 20:13:54.857655 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0923 20:13:54.859333 (+ 1678us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:13:54.859937 (+ 604us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:13:54.859944 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:13:54.860021 (+ 77us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:13:54.860531 (+ 510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:13:54.860546 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:13:54.861529 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:13:54.861533 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:13:54.863233 (+ 1700us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:13:54.863249 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:13:54.864039 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:13:54.864046 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:13:54.864275 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:13:54.864997 (+ 722us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 20:13:54.865000 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:13:54.865057 (+ 57us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 20:13:54.865151 (+ 94us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:13:54.865533 (+ 382us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:13:54.865541 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:13:54.865546 (+ 5us) client_negotiation.cc:769] Sending connection context
0923 20:13:54.865680 (+ 134us) client_negotiation.cc:240] Negotiation successful
0923 20:13:54.866024 (+ 344us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":449,"thread_start_us":278,"threads_started":1}
I20260923 20:13:54.868360 19909 heartbeater.cc:347] Connected to a master server at 127.1.170.190:42127
I20260923 20:13:54.868656 19909 heartbeater.cc:464] Registering TS with master...
I20260923 20:13:54.869320 19909 heartbeater.cc:511] Master 127.1.170.190:42127 requested a full tablet report, sending...
I20260923 20:13:54.870615 19462 ts_manager.cc:194] Registered new tserver with Master: 1c2285e6a7914d55bb93642018d06852 (127.1.170.131:46035)
I20260923 20:13:54.871383 19462 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.170.131:55843
I20260923 20:13:54.878418 1706 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260923 20:13:54.888389 19650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:13:54.882385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52114 (local address 127.1.170.190:42127)
0923 20:13:54.882615 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0923 20:13:54.882623 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 20:13:54.883019 (+ 396us) server_negotiation.cc:407] Connection header received
0923 20:13:54.883211 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:13:54.883216 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:13:54.883292 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:13:54.883392 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:13:54.883890 (+ 498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:13:54.884592 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:13:54.886087 (+ 1495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:13:54.886476 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:13:54.886610 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:13:54.887695 (+ 1085us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:13:54.887700 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:13:54.887703 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 20:13:54.887719 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:13:54.887749 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 20:13:54.887760 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:13:54.887853 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0923 20:13:54.888016 (+ 163us) server_negotiation.cc:299] Negotiation successful
0923 20:13:54.888104 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260923 20:13:54.891918 19462 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52114:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 20:13:54.898614 19462 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 20:13:54.938675 19924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:13:54.910595 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55684 (local address 127.1.170.129:43077)
0923 20:13:54.924823 (+ 14228us) server_negotiation.cc:206] Beginning negotiation
0923 20:13:54.924837 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0923 20:13:54.925430 (+ 593us) server_negotiation.cc:407] Connection header received
0923 20:13:54.925778 (+ 348us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:13:54.925787 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:13:54.926112 (+ 325us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:13:54.926330 (+ 218us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:13:54.928879 (+ 2549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:13:54.932746 (+ 3867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:13:54.935988 (+ 3242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:13:54.937293 (+ 1305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:13:54.937455 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:13:54.937764 (+ 309us) server_negotiation.cc:1035] Waiting for connection context
0923 20:13:54.938114 (+ 350us) server_negotiation.cc:299] Negotiation successful
0923 20:13:54.938291 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14027,"thread_start_us":13880,"threads_started":1}
I20260923 20:13:54.939692 19921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:13:54.910303 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.129:43077 (local address 127.0.0.1:55684)
0923 20:13:54.925317 (+ 15014us) negotiation.cc:107] Waiting for socket to connect
0923 20:13:54.925337 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0923 20:13:54.925559 (+ 222us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:13:54.926785 (+ 1226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:13:54.926793 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:13:54.926828 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:13:54.928601 (+ 1773us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:13:54.928621 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:13:54.932980 (+ 4359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:13:54.932984 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:13:54.935748 (+ 2764us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:13:54.935763 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:13:54.937505 (+ 1742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:13:54.937512 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:13:54.937749 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:13:54.937811 (+ 62us) client_negotiation.cc:769] Sending connection context
0923 20:13:54.939064 (+ 1253us) client_negotiation.cc:240] Negotiation successful
0923 20:13:54.939282 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14876,"mutex_wait_us":1,"thread_start_us":12083,"threads_started":1}
I20260923 20:13:54.940019 19923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:13:54.910350 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56050 (local address 127.1.170.130:38621)
0923 20:13:54.932688 (+ 22338us) server_negotiation.cc:206] Beginning negotiation
0923 20:13:54.932699 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0923 20:13:54.932750 (+ 51us) server_negotiation.cc:407] Connection header received
0923 20:13:54.932863 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:13:54.932870 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:13:54.933144 (+ 274us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:13:54.933409 (+ 265us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:13:54.934438 (+ 1029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:13:54.935881 (+ 1443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:13:54.937929 (+ 2048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:13:54.938872 (+ 943us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:13:54.939004 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:13:54.939248 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0923 20:13:54.939502 (+ 254us) server_negotiation.cc:299] Negotiation successful
0923 20:13:54.939712 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22188,"thread_start_us":22089,"threads_started":1}
I20260923 20:13:54.940235 19922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:13:54.910184 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.130:38621 (local address 127.0.0.1:56050)
0923 20:13:54.922819 (+ 12635us) negotiation.cc:107] Waiting for socket to connect
0923 20:13:54.922869 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0923 20:13:54.923067 (+ 198us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:13:54.933530 (+ 10463us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:13:54.933535 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:13:54.933566 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:13:54.934217 (+ 651us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:13:54.934230 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:13:54.936054 (+ 1824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:13:54.936062 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:13:54.937725 (+ 1663us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:13:54.937737 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:13:54.939127 (+ 1390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:13:54.939134 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:13:54.939329 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:13:54.939352 (+ 23us) client_negotiation.cc:769] Sending connection context
0923 20:13:54.939454 (+ 102us) client_negotiation.cc:240] Negotiation successful
0923 20:13:54.939596 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12332,"mutex_wait_us":36,"thread_start_us":14826,"threads_started":1}
I20260923 20:13:54.942801 19925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:13:54.925718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.131:46035 (local address 127.0.0.1:35530)
0923 20:13:54.933038 (+ 7320us) negotiation.cc:107] Waiting for socket to connect
0923 20:13:54.933056 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0923 20:13:54.933199 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:13:54.933829 (+ 630us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:13:54.933836 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:13:54.933866 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:13:54.934477 (+ 611us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:13:54.934486 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:13:54.935986 (+ 1500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:13:54.935992 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:13:54.939604 (+ 3612us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:13:54.939625 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:13:54.942044 (+ 2419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:13:54.942054 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:13:54.942236 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:13:54.942266 (+ 30us) client_negotiation.cc:769] Sending connection context
0923 20:13:54.942352 (+ 86us) client_negotiation.cc:240] Negotiation successful
0923 20:13:54.942476 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7183,"mutex_wait_us":29,"thread_start_us":7041,"threads_started":1}
I20260923 20:13:54.942960 19926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:13:54.925807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35530 (local address 127.1.170.131:46035)
0923 20:13:54.932238 (+ 6431us) server_negotiation.cc:206] Beginning negotiation
0923 20:13:54.932250 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0923 20:13:54.933194 (+ 944us) server_negotiation.cc:407] Connection header received
0923 20:13:54.933352 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:13:54.933361 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:13:54.933575 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:13:54.933761 (+ 186us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:13:54.934657 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:13:54.935813 (+ 1156us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:13:54.940155 (+ 4342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:13:54.941558 (+ 1403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:13:54.941701 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:13:54.941993 (+ 292us) server_negotiation.cc:1035] Waiting for connection context
0923 20:13:54.942444 (+ 451us) server_negotiation.cc:299] Negotiation successful
0923 20:13:54.942607 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6217,"thread_start_us":6079,"threads_started":1}
I20260923 20:13:54.944084 19714 tablet_service.cc:1467] Processing CreateTablet for tablet 3d7ee87f511843a1a7c2c682acac4ba0 (DEFAULT_TABLE table=test-table [id=d3a0b02a5d3f441eb13550b1703338d8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 20:13:54.944080 19583 tablet_service.cc:1467] Processing CreateTablet for tablet 3d7ee87f511843a1a7c2c682acac4ba0 (DEFAULT_TABLE table=test-table [id=d3a0b02a5d3f441eb13550b1703338d8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 20:13:54.945444 19714 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3d7ee87f511843a1a7c2c682acac4ba0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:13:54.945606 19583 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3d7ee87f511843a1a7c2c682acac4ba0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:13:54.946399 19844 tablet_service.cc:1467] Processing CreateTablet for tablet 3d7ee87f511843a1a7c2c682acac4ba0 (DEFAULT_TABLE table=test-table [id=d3a0b02a5d3f441eb13550b1703338d8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 20:13:54.947347 19648 heartbeater.cc:503] Master 127.1.170.190:42127 was elected leader, sending a full tablet report...
I20260923 20:13:54.949007 19844 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3d7ee87f511843a1a7c2c682acac4ba0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:13:54.957217 19927 tablet_bootstrap.cc:492] T 3d7ee87f511843a1a7c2c682acac4ba0 P fbcc86fecac3454fbd3568c27f3d26a6: Bootstrap starting.
I20260923 20:13:54.960585 19927 tablet_bootstrap.cc:654] T 3d7ee87f511843a1a7c2c682acac4ba0 P fbcc86fecac3454fbd3568c27f3d26a6: Neither blocks nor log segments found. Creating new log.
I20260923 20:13:54.961565 19927 log.cc:824] T 3d7ee87f511843a1a7c2c682acac4ba0 P fbcc86fecac3454fbd3568c27f3d26a6: Log is configured to *not* fsync() on all Append() calls
I20260923 20:13:54.962580 19928 tablet_bootstrap.cc:492] T 3d7ee87f511843a1a7c2c682acac4ba0 P 212456b209c24be897b2edbfd782bdc4: Bootstrap starting.
I20260923 20:13:54.965821 19930 tablet_bootstrap.cc:492] T 3d7ee87f511843a1a7c2c682acac4ba0 P 1c2285e6a7914d55bb93642018d06852: Bootstrap starting.
I20260923 20:13:54.966656 19928 tablet_bootstrap.cc:654] T 3d7ee87f511843a1a7c2c682acac4ba0 P 212456b209c24be897b2edbfd782bdc4: Neither blocks nor log segments found. Creating new log.
I20260923 20:13:54.967060 19927 tablet_bootstrap.cc:492] T 3d7ee87f511843a1a7c2c682acac4ba0 P fbcc86fecac3454fbd3568c27f3d26a6: No bootstrap required, opened a new log
I20260923 20:13:54.967325 19927 ts_tablet_manager.cc:1397] T 3d7ee87f511843a1a7c2c682acac4ba0 P fbcc86fecac3454fbd3568c27f3d26a6: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260923 20:13:54.967885 19928 log.cc:824] T 3d7ee87f511843a1a7c2c682acac4ba0 P 212456b209c24be897b2edbfd782bdc4: Log is configured to *not* fsync() on all Append() calls
I20260923 20:13:54.968699 19930 tablet_bootstrap.cc:654] T 3d7ee87f511843a1a7c2c682acac4ba0 P 1c2285e6a7914d55bb93642018d06852: Neither blocks nor log segments found. Creating new log.
I20260923 20:13:54.969725 19930 log.cc:824] T 3d7ee87f511843a1a7c2c682acac4ba0 P 1c2285e6a7914d55bb93642018d06852: Log is configured to *not* fsync() on all Append() calls
I20260923 20:13:54.970981 19928 tablet_bootstrap.cc:492] T 3d7ee87f511843a1a7c2c682acac4ba0 P 212456b209c24be897b2edbfd782bdc4: No bootstrap required, opened a new log
I20260923 20:13:54.971244 19928 ts_tablet_manager.cc:1397] T 3d7ee87f511843a1a7c2c682acac4ba0 P 212456b209c24be897b2edbfd782bdc4: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.003s
I20260923 20:13:54.972631 19930 tablet_bootstrap.cc:492] T 3d7ee87f511843a1a7c2c682acac4ba0 P 1c2285e6a7914d55bb93642018d06852: No bootstrap required, opened a new log
I20260923 20:13:54.972890 19930 ts_tablet_manager.cc:1397] T 3d7ee87f511843a1a7c2c682acac4ba0 P 1c2285e6a7914d55bb93642018d06852: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.003s
I20260923 20:13:54.977794 19927 raft_consensus.cc:348] T 3d7ee87f511843a1a7c2c682acac4ba0 P fbcc86fecac3454fbd3568c27f3d26a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbcc86fecac3454fbd3568c27f3d26a6" member_type: VOTER last_known_addr { host: "127.1.170.130" port: 38621 } } peers { permanent_uuid: "212456b209c24be897b2edbfd782bdc4" member_type: VOTER last_known_addr { host: "127.1.170.129" port: 43077 } } peers { permanent_uuid: "1c2285e6a7914d55bb93642018d06852" member_type: VOTER last_known_addr { host: "127.1.170.131" port: 46035 } }
I20260923 20:13:54.978307 19927 raft_consensus.cc:374] T 3d7ee87f511843a1a7c2c682acac4ba0 P fbcc86fecac3454fbd3568c27f3d26a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:13:54.978407 19927 raft_consensus.cc:729] T 3d7ee87f511843a1a7c2c682acac4ba0 P fbcc86fecac3454fbd3568c27f3d26a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fbcc86fecac3454fbd3568c27f3d26a6, State: Initialized, Role: FOLLOWER
I20260923 20:13:54.979076 19927 consensus_queue.cc:260] T 3d7ee87f511843a1a7c2c682acac4ba0 P fbcc86fecac3454fbd3568c27f3d26a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbcc86fecac3454fbd3568c27f3d26a6" member_type: VOTER last_known_addr { host: "127.1.170.130" port: 38621 } } peers { permanent_uuid: "212456b209c24be897b2edbfd782bdc4" member_type: VOTER last_known_addr { host: "127.1.170.129" port: 43077 } } peers { permanent_uuid: "1c2285e6a7914d55bb93642018d06852" member_type: VOTER last_known_addr { host: "127.1.170.131" port: 46035 } }
I20260923 20:13:54.980391 19779 heartbeater.cc:503] Master 127.1.170.190:42127 was elected leader, sending a full tablet report...
I20260923 20:13:54.981189 19927 ts_tablet_manager.cc:1428] T 3d7ee87f511843a1a7c2c682acac4ba0 P fbcc86fecac3454fbd3568c27f3d26a6: Time spent starting tablet: real 0.014s user 0.015s sys 0.000s
I20260923 20:13:54.981140 19930 raft_consensus.cc:348] T 3d7ee87f511843a1a7c2c682acac4ba0 P 1c2285e6a7914d55bb93642018d06852 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbcc86fecac3454fbd3568c27f3d26a6" member_type: VOTER last_known_addr { host: "127.1.170.130" port: 38621 } } peers { permanent_uuid: "212456b209c24be897b2edbfd782bdc4" member_type: VOTER last_known_addr { host: "127.1.170.129" port: 43077 } } peers { permanent_uuid: "1c2285e6a7914d55bb93642018d06852" member_type: VOTER last_known_addr { host: "127.1.170.131" port: 46035 } }
I20260923 20:13:54.981578 19930 raft_consensus.cc:374] T 3d7ee87f511843a1a7c2c682acac4ba0 P 1c2285e6a7914d55bb93642018d06852 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:13:54.981691 19930 raft_consensus.cc:729] T 3d7ee87f511843a1a7c2c682acac4ba0 P 1c2285e6a7914d55bb93642018d06852 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c2285e6a7914d55bb93642018d06852, State: Initialized, Role: FOLLOWER
I20260923 20:13:54.982033 19928 raft_consensus.cc:348] T 3d7ee87f511843a1a7c2c682acac4ba0 P 212456b209c24be897b2edbfd782bdc4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbcc86fecac3454fbd3568c27f3d26a6" member_type: VOTER last_known_addr { host: "127.1.170.130" port: 38621 } } peers { permanent_uuid: "212456b209c24be897b2edbfd782bdc4" member_type: VOTER last_known_addr { host: "127.1.170.129" port: 43077 } } peers { permanent_uuid: "1c2285e6a7914d55bb93642018d06852" member_type: VOTER last_known_addr { host: "127.1.170.131" port: 46035 } }
I20260923 20:13:54.982477 19928 raft_consensus.cc:374] T 3d7ee87f511843a1a7c2c682acac4ba0 P 212456b209c24be897b2edbfd782bdc4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:13:54.982342 19930 consensus_queue.cc:260] T 3d7ee87f511843a1a7c2c682acac4ba0 P 1c2285e6a7914d55bb93642018d06852 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbcc86fecac3454fbd3568c27f3d26a6" member_type: VOTER last_known_addr { host: "127.1.170.130" port: 38621 } } peers { permanent_uuid: "212456b209c24be897b2edbfd782bdc4" member_type: VOTER last_known_addr { host: "127.1.170.129" port: 43077 } } peers { permanent_uuid: "1c2285e6a7914d55bb93642018d06852" member_type: VOTER last_known_addr { host: "127.1.170.131" port: 46035 } }
I20260923 20:13:54.982580 19928 raft_consensus.cc:729] T 3d7ee87f511843a1a7c2c682acac4ba0 P 212456b209c24be897b2edbfd782bdc4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 212456b209c24be897b2edbfd782bdc4, State: Initialized, Role: FOLLOWER
I20260923 20:13:54.983124 19928 consensus_queue.cc:260] T 3d7ee87f511843a1a7c2c682acac4ba0 P 212456b209c24be897b2edbfd782bdc4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbcc86fecac3454fbd3568c27f3d26a6" member_type: VOTER last_known_addr { host: "127.1.170.130" port: 38621 } } peers { permanent_uuid: "212456b209c24be897b2edbfd782bdc4" member_type: VOTER last_known_addr { host: "127.1.170.129" port: 43077 } } peers { permanent_uuid: "1c2285e6a7914d55bb93642018d06852" member_type: VOTER last_known_addr { host: "127.1.170.131" port: 46035 } }
I20260923 20:13:54.985018 19928 ts_tablet_manager.cc:1428] T 3d7ee87f511843a1a7c2c682acac4ba0 P 212456b209c24be897b2edbfd782bdc4: Time spent starting tablet: real 0.014s user 0.014s sys 0.000s
I20260923 20:13:54.985330 19909 heartbeater.cc:503] Master 127.1.170.190:42127 was elected leader, sending a full tablet report...
I20260923 20:13:54.986018 19930 ts_tablet_manager.cc:1428] T 3d7ee87f511843a1a7c2c682acac4ba0 P 1c2285e6a7914d55bb93642018d06852: Time spent starting tablet: real 0.013s user 0.013s sys 0.001s
I20260923 20:13:55.063046 19934 raft_consensus.cc:482] T 3d7ee87f511843a1a7c2c682acac4ba0 P 1c2285e6a7914d55bb93642018d06852 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 20:13:55.063335 19934 raft_consensus.cc:504] T 3d7ee87f511843a1a7c2c682acac4ba0 P 1c2285e6a7914d55bb93642018d06852 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbcc86fecac3454fbd3568c27f3d26a6" member_type: VOTER last_known_addr { host: "127.1.170.130" port: 38621 } } peers { permanent_uuid: "212456b209c24be897b2edbfd782bdc4" member_type: VOTER last_known_addr { host: "127.1.170.129" port: 43077 } } peers { permanent_uuid: "1c2285e6a7914d55bb93642018d06852" member_type: VOTER last_known_addr { host: "127.1.170.131" port: 46035 } }
I20260923 20:13:55.064968 19934 leader_election.cc:288] T 3d7ee87f511843a1a7c2c682acac4ba0 P 1c2285e6a7914d55bb93642018d06852 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fbcc86fecac3454fbd3568c27f3d26a6 (127.1.170.130:38621), 212456b209c24be897b2edbfd782bdc4 (127.1.170.129:43077)
I20260923 20:13:55.071714 19923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:13:55.065375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.170.131:45729 (local address 127.1.170.130:38621)
0923 20:13:55.065577 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0923 20:13:55.065585 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 20:13:55.065940 (+ 355us) server_negotiation.cc:407] Connection header received
0923 20:13:55.066235 (+ 295us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:13:55.066241 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:13:55.066335 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:13:55.066432 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:13:55.066905 (+ 473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:13:55.067803 (+ 898us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:13:55.069723 (+ 1920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:13:55.070780 (+ 1057us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:13:55.070944 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:13:55.070996 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0923 20:13:55.071312 (+ 316us) server_negotiation.cc:299] Negotiation successful
0923 20:13:55.071446 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260923 20:13:55.071771 19911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:13:55.065435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.130:38621 (local address 127.1.170.131:45729)
0923 20:13:55.065751 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0923 20:13:55.065777 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0923 20:13:55.066054 (+ 277us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:13:55.066521 (+ 467us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:13:55.066529 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:13:55.066559 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:13:55.066746 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:13:55.066760 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:13:55.067968 (+ 1208us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:13:55.067974 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:13:55.069559 (+ 1585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:13:55.069571 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:13:55.070951 (+ 1380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:13:55.070957 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:13:55.071135 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:13:55.071171 (+ 36us) client_negotiation.cc:769] Sending connection context
0923 20:13:55.071268 (+ 97us) client_negotiation.cc:240] Negotiation successful
0923 20:13:55.071421 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":184}
I20260923 20:13:55.072175 19924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:13:55.065711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.170.131:37843 (local address 127.1.170.129:43077)
0923 20:13:55.065989 (+ 278us) server_negotiation.cc:206] Beginning negotiation
0923 20:13:55.065998 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 20:13:55.066442 (+ 444us) server_negotiation.cc:407] Connection header received
0923 20:13:55.066597 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:13:55.066602 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:13:55.066684 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:13:55.066784 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:13:55.067418 (+ 634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:13:55.068350 (+ 932us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:13:55.070280 (+ 1930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:13:55.071311 (+ 1031us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:13:55.071463 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:13:55.071528 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0923 20:13:55.071821 (+ 293us) server_negotiation.cc:299] Negotiation successful
0923 20:13:55.071950 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260923 20:13:55.072619 19936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:13:55.065502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.129:43077 (local address 127.1.170.131:37843)
0923 20:13:55.066317 (+ 815us) negotiation.cc:107] Waiting for socket to connect
0923 20:13:55.066333 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0923 20:13:55.066458 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:13:55.066808 (+ 350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:13:55.066812 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:13:55.066855 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:13:55.067233 (+ 378us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:13:55.067245 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:13:55.068544 (+ 1299us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:13:55.068548 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:13:55.070100 (+ 1552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:13:55.070114 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:13:55.071466 (+ 1352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:13:55.071472 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:13:55.071642 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:13:55.071672 (+ 30us) client_negotiation.cc:769] Sending connection context
0923 20:13:55.072165 (+ 493us) client_negotiation.cc:240] Negotiation successful
0923 20:13:55.072353 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":683,"thread_start_us":440,"threads_started":1}
I20260923 20:13:55.073043 19734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d7ee87f511843a1a7c2c682acac4ba0" candidate_uuid: "1c2285e6a7914d55bb93642018d06852" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fbcc86fecac3454fbd3568c27f3d26a6" is_pre_election: true
I20260923 20:13:55.073630 19734 raft_consensus.cc:2454] T 3d7ee87f511843a1a7c2c682acac4ba0 P fbcc86fecac3454fbd3568c27f3d26a6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c2285e6a7914d55bb93642018d06852 in term 0.
I20260923 20:13:55.073727 19603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d7ee87f511843a1a7c2c682acac4ba0" candidate_uuid: "1c2285e6a7914d55bb93642018d06852" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "212456b209c24be897b2edbfd782bdc4" is_pre_election: true
I20260923 20:13:55.074349 19603 raft_consensus.cc:2454] T 3d7ee87f511843a1a7c2c682acac4ba0 P 212456b209c24be897b2edbfd782bdc4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c2285e6a7914d55bb93642018d06852 in term 0.
I20260923 20:13:55.074378 19799 leader_election.cc:302] T 3d7ee87f511843a1a7c2c682acac4ba0 P 1c2285e6a7914d55bb93642018d06852 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c2285e6a7914d55bb93642018d06852, fbcc86fecac3454fbd3568c27f3d26a6; no voters:
I20260923 20:13:55.074813 19934 raft_consensus.cc:2781] T 3d7ee87f511843a1a7c2c682acac4ba0 P 1c2285e6a7914d55bb93642018d06852 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 20:13:55.074961 19934 raft_consensus.cc:482] T 3d7ee87f511843a1a7c2c682acac4ba0 P 1c2285e6a7914d55bb93642018d06852 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 20:13:55.075078 19934 raft_consensus.cc:3037] T 3d7ee87f511843a1a7c2c682acac4ba0 P 1c2285e6a7914d55bb93642018d06852 [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:13:55.077251 19934 raft_consensus.cc:504] T 3d7ee87f511843a1a7c2c682acac4ba0 P 1c2285e6a7914d55bb93642018d06852 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbcc86fecac3454fbd3568c27f3d26a6" member_type: VOTER last_known_addr { host: "127.1.170.130" port: 38621 } } peers { permanent_uuid: "212456b209c24be897b2edbfd782bdc4" member_type: VOTER last_known_addr { host: "127.1.170.129" port: 43077 } } peers { permanent_uuid: "1c2285e6a7914d55bb93642018d06852" member_type: VOTER last_known_addr { host: "127.1.170.131" port: 46035 } }
I20260923 20:13:55.077915 19934 leader_election.cc:288] T 3d7ee87f511843a1a7c2c682acac4ba0 P 1c2285e6a7914d55bb93642018d06852 [CANDIDATE]: Term 1 election: Requested vote from peers fbcc86fecac3454fbd3568c27f3d26a6 (127.1.170.130:38621), 212456b209c24be897b2edbfd782bdc4 (127.1.170.129:43077)
I20260923 20:13:55.078431 19734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d7ee87f511843a1a7c2c682acac4ba0" candidate_uuid: "1c2285e6a7914d55bb93642018d06852" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fbcc86fecac3454fbd3568c27f3d26a6"
I20260923 20:13:55.078482 19603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d7ee87f511843a1a7c2c682acac4ba0" candidate_uuid: "1c2285e6a7914d55bb93642018d06852" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "212456b209c24be897b2edbfd782bdc4"
I20260923 20:13:55.078670 19734 raft_consensus.cc:3037] T 3d7ee87f511843a1a7c2c682acac4ba0 P fbcc86fecac3454fbd3568c27f3d26a6 [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:13:55.078691 19603 raft_consensus.cc:3037] T 3d7ee87f511843a1a7c2c682acac4ba0 P 212456b209c24be897b2edbfd782bdc4 [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:13:55.082057 19603 raft_consensus.cc:2454] T 3d7ee87f511843a1a7c2c682acac4ba0 P 212456b209c24be897b2edbfd782bdc4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c2285e6a7914d55bb93642018d06852 in term 1.
I20260923 20:13:55.082057 19734 raft_consensus.cc:2454] T 3d7ee87f511843a1a7c2c682acac4ba0 P fbcc86fecac3454fbd3568c27f3d26a6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c2285e6a7914d55bb93642018d06852 in term 1.
I20260923 20:13:55.082484 19800 leader_election.cc:302] T 3d7ee87f511843a1a7c2c682acac4ba0 P 1c2285e6a7914d55bb93642018d06852 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c2285e6a7914d55bb93642018d06852, 212456b209c24be897b2edbfd782bdc4; no voters:
I20260923 20:13:55.082823 19934 raft_consensus.cc:2781] T 3d7ee87f511843a1a7c2c682acac4ba0 P 1c2285e6a7914d55bb93642018d06852 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 20:13:55.083284 19934 raft_consensus.cc:686] T 3d7ee87f511843a1a7c2c682acac4ba0 P 1c2285e6a7914d55bb93642018d06852 [term 1 LEADER]: Becoming Leader. State: Replica: 1c2285e6a7914d55bb93642018d06852, State: Running, Role: LEADER
I20260923 20:13:55.083748 19934 consensus_queue.cc:237] T 3d7ee87f511843a1a7c2c682acac4ba0 P 1c2285e6a7914d55bb93642018d06852 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbcc86fecac3454fbd3568c27f3d26a6" member_type: VOTER last_known_addr { host: "127.1.170.130" port: 38621 } } peers { permanent_uuid: "212456b209c24be897b2edbfd782bdc4" member_type: VOTER last_known_addr { host: "127.1.170.129" port: 43077 } } peers { permanent_uuid: "1c2285e6a7914d55bb93642018d06852" member_type: VOTER last_known_addr { host: "127.1.170.131" port: 46035 } }
I20260923 20:13:55.088478 19462 catalog_manager.cc:5495] T 3d7ee87f511843a1a7c2c682acac4ba0 P 1c2285e6a7914d55bb93642018d06852 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1c2285e6a7914d55bb93642018d06852 (127.1.170.131). New cstate: current_term: 1 leader_uuid: "1c2285e6a7914d55bb93642018d06852" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbcc86fecac3454fbd3568c27f3d26a6" member_type: VOTER last_known_addr { host: "127.1.170.130" port: 38621 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "212456b209c24be897b2edbfd782bdc4" member_type: VOTER last_known_addr { host: "127.1.170.129" port: 43077 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1c2285e6a7914d55bb93642018d06852" member_type: VOTER last_known_addr { host: "127.1.170.131" port: 46035 } health_report { overall_health: HEALTHY } } }
W20260923 20:13:55.105290 19910 tablet.cc:2369] T 3d7ee87f511843a1a7c2c682acac4ba0 P 1c2285e6a7914d55bb93642018d06852: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 20:13:55.128788 19926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:13:55.123101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35538 (local address 127.1.170.131:46035)
0923 20:13:55.123322 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0923 20:13:55.123329 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:13:55.123472 (+ 143us) server_negotiation.cc:407] Connection header received
0923 20:13:55.123682 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:13:55.123686 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:13:55.123767 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:13:55.123895 (+ 128us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 20:13:55.124301 (+ 406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:13:55.125030 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:13:55.126577 (+ 1547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:13:55.127010 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:13:55.127109 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:13:55.127533 (+ 424us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 20:13:55.128085 (+ 552us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 20:13:55.128197 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0923 20:13:55.128386 (+ 189us) server_negotiation.cc:299] Negotiation successful
0923 20:13:55.128490 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260923 20:13:55.137133 19603 raft_consensus.cc:1261] T 3d7ee87f511843a1a7c2c682acac4ba0 P 212456b209c24be897b2edbfd782bdc4 [term 1 FOLLOWER]: Refusing update from remote peer 1c2285e6a7914d55bb93642018d06852: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 20:13:55.137132 19734 raft_consensus.cc:1261] T 3d7ee87f511843a1a7c2c682acac4ba0 P fbcc86fecac3454fbd3568c27f3d26a6 [term 1 FOLLOWER]: Refusing update from remote peer 1c2285e6a7914d55bb93642018d06852: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 20:13:55.138139 19934 consensus_queue.cc:1035] T 3d7ee87f511843a1a7c2c682acac4ba0 P 1c2285e6a7914d55bb93642018d06852 [LEADER]: Connected to new peer: Peer: permanent_uuid: "212456b209c24be897b2edbfd782bdc4" member_type: VOTER last_known_addr { host: "127.1.170.129" port: 43077 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 20:13:55.138447 19937 consensus_queue.cc:1035] T 3d7ee87f511843a1a7c2c682acac4ba0 P 1c2285e6a7914d55bb93642018d06852 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fbcc86fecac3454fbd3568c27f3d26a6" member_type: VOTER last_known_addr { host: "127.1.170.130" port: 38621 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260923 20:13:55.149089 19780 tablet.cc:2369] T 3d7ee87f511843a1a7c2c682acac4ba0 P fbcc86fecac3454fbd3568c27f3d26a6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 20:13:55.149992 19940 mvcc.cc:205] Tried to move back new op lower bound from 7332636406312103936 to 7332636406107324416. Current Snapshot: MvccSnapshot[applied={T|T < 7332636406312103936}]
I20260923 20:13:55.151386 19942 mvcc.cc:205] Tried to move back new op lower bound from 7332636406312103936 to 7332636406107324416. Current Snapshot: MvccSnapshot[applied={T|T < 7332636406312103936}]
I20260923 20:13:55.165295 19462 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:52114:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 20:13:55.165596 19462 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:52114:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 20:13:55.169634 19462 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 380fc491ed614a1782495140d75136d0: Sending DeleteTablet for 3 replicas of tablet 3d7ee87f511843a1a7c2c682acac4ba0
I20260923 20:13:55.171033 19844 tablet_service.cc:1514] Processing DeleteTablet for tablet 3d7ee87f511843a1a7c2c682acac4ba0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 20:13:55 UTC) from {username='slave'} at 127.0.0.1:35530
I20260923 20:13:55.170985 19583 tablet_service.cc:1514] Processing DeleteTablet for tablet 3d7ee87f511843a1a7c2c682acac4ba0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 20:13:55 UTC) from {username='slave'} at 127.0.0.1:55684
I20260923 20:13:55.171033 19714 tablet_service.cc:1514] Processing DeleteTablet for tablet 3d7ee87f511843a1a7c2c682acac4ba0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 20:13:55 UTC) from {username='slave'} at 127.0.0.1:56050
I20260923 20:13:55.172641 1706 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskqUuku6/build/asan/bin/kudu pid 19521
I20260923 20:13:55.183583 19950 tablet_replica.cc:330] T 3d7ee87f511843a1a7c2c682acac4ba0 P 1c2285e6a7914d55bb93642018d06852: stopping tablet replica
I20260923 20:13:55.184123 19950 raft_consensus.cc:2229] T 3d7ee87f511843a1a7c2c682acac4ba0 P 1c2285e6a7914d55bb93642018d06852 [term 1 LEADER]: Raft consensus shutting down.
I20260923 20:13:55.184863 19950 raft_consensus.cc:2258] T 3d7ee87f511843a1a7c2c682acac4ba0 P 1c2285e6a7914d55bb93642018d06852 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 20:13:55.185660 19951 tablet_replica.cc:330] T 3d7ee87f511843a1a7c2c682acac4ba0 P 212456b209c24be897b2edbfd782bdc4: stopping tablet replica
I20260923 20:13:55.186107 19951 raft_consensus.cc:2229] T 3d7ee87f511843a1a7c2c682acac4ba0 P 212456b209c24be897b2edbfd782bdc4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 20:13:55.186654 19951 raft_consensus.cc:2258] T 3d7ee87f511843a1a7c2c682acac4ba0 P 212456b209c24be897b2edbfd782bdc4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 20:13:55.187690 19949 tablet_replica.cc:330] T 3d7ee87f511843a1a7c2c682acac4ba0 P fbcc86fecac3454fbd3568c27f3d26a6: stopping tablet replica
I20260923 20:13:55.188258 19949 raft_consensus.cc:2229] T 3d7ee87f511843a1a7c2c682acac4ba0 P fbcc86fecac3454fbd3568c27f3d26a6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 20:13:55.188719 19949 raft_consensus.cc:2258] T 3d7ee87f511843a1a7c2c682acac4ba0 P fbcc86fecac3454fbd3568c27f3d26a6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 20:13:55.189294 19950 ts_tablet_manager.cc:1905] T 3d7ee87f511843a1a7c2c682acac4ba0 P 1c2285e6a7914d55bb93642018d06852: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 20:13:55.192698 19949 ts_tablet_manager.cc:1905] T 3d7ee87f511843a1a7c2c682acac4ba0 P fbcc86fecac3454fbd3568c27f3d26a6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 20:13:55.193620 19924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:13:55.174215 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55686 (local address 127.1.170.129:43077)
0923 20:13:55.174473 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0923 20:13:55.174483 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0923 20:13:55.184264 (+ 9781us) server_negotiation.cc:407] Connection header received
0923 20:13:55.184685 (+ 421us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:13:55.184693 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:13:55.184808 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:13:55.184913 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:13:55.185716 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:13:55.187008 (+ 1292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:13:55.189313 (+ 2305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:13:55.190053 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:13:55.190152 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:13:55.191893 (+ 1741us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:13:55.191908 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:13:55.191924 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 20:13:55.192653 (+ 729us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:13:55.192718 (+ 65us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 20:13:55.192765 (+ 47us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:13:55.192870 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0923 20:13:55.193165 (+ 295us) server_negotiation.cc:299] Negotiation successful
0923 20:13:55.193287 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260923 20:13:55.194996 19643 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:13:55.196885 19950 ts_tablet_manager.cc:1918] T 3d7ee87f511843a1a7c2c682acac4ba0 P 1c2285e6a7914d55bb93642018d06852: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 20:13:55.197058 19950 log.cc:1196] T 3d7ee87f511843a1a7c2c682acac4ba0 P 1c2285e6a7914d55bb93642018d06852: Deleting WAL directory at /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790194121577027-1706-0/minicluster-data/ts-2/wal/wals/3d7ee87f511843a1a7c2c682acac4ba0
I20260923 20:13:55.197602 19950 ts_tablet_manager.cc:1939] T 3d7ee87f511843a1a7c2c682acac4ba0 P 1c2285e6a7914d55bb93642018d06852: Deleting consensus metadata
I20260923 20:13:55.199548 19448 catalog_manager.cc:4841] TS 1c2285e6a7914d55bb93642018d06852 (127.1.170.131:46035): tablet 3d7ee87f511843a1a7c2c682acac4ba0 (table test-table [id=d3a0b02a5d3f441eb13550b1703338d8]) successfully deleted
I20260923 20:13:55.201534 19949 ts_tablet_manager.cc:1918] T 3d7ee87f511843a1a7c2c682acac4ba0 P fbcc86fecac3454fbd3568c27f3d26a6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 20:13:55.201890 19949 log.cc:1196] T 3d7ee87f511843a1a7c2c682acac4ba0 P fbcc86fecac3454fbd3568c27f3d26a6: Deleting WAL directory at /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790194121577027-1706-0/minicluster-data/ts-1/wal/wals/3d7ee87f511843a1a7c2c682acac4ba0
I20260923 20:13:55.202975 19949 ts_tablet_manager.cc:1939] T 3d7ee87f511843a1a7c2c682acac4ba0 P fbcc86fecac3454fbd3568c27f3d26a6: Deleting consensus metadata
I20260923 20:13:55.204758 19448 catalog_manager.cc:4841] TS fbcc86fecac3454fbd3568c27f3d26a6 (127.1.170.130:38621): tablet 3d7ee87f511843a1a7c2c682acac4ba0 (table test-table [id=d3a0b02a5d3f441eb13550b1703338d8]) successfully deleted
I20260923 20:13:55.190744 19951 ts_tablet_manager.cc:1905] T 3d7ee87f511843a1a7c2c682acac4ba0 P 212456b209c24be897b2edbfd782bdc4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 20:13:55.307649 1706 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqUuku6/build/asan/bin/kudu with pid 19521
I20260923 20:13:55.325897 1706 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskqUuku6/build/asan/bin/kudu pid 19652
W20260923 20:13:55.325910 19449 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.1.170.129:43077 (error 108)
I20260923 20:13:55.328517 19925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:13:55.327927 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.129:43077 (local address 127.0.0.1:55690)
0923 20:13:55.328170 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0923 20:13:55.328314 (+ 144us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.170.129:43077: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133}
W20260923 20:13:55.328795 19449 catalog_manager.cc:4584] TS 212456b209c24be897b2edbfd782bdc4 (127.1.170.129:43077): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 3d7ee87f511843a1a7c2c682acac4ba0: Network error: Client connection negotiation failed: client connection to 127.1.170.129:43077: connect: Connection refused (error 111)
I20260923 20:13:55.333966 19923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:13:55.327051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56064 (local address 127.1.170.130:38621)
0923 20:13:55.327233 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0923 20:13:55.327239 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:13:55.327260 (+ 21us) server_negotiation.cc:407] Connection header received
0923 20:13:55.327328 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:13:55.327333 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:13:55.327462 (+ 129us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:13:55.327585 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:13:55.328192 (+ 607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:13:55.329298 (+ 1106us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:13:55.330886 (+ 1588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:13:55.331281 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:13:55.331418 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:13:55.332531 (+ 1113us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:13:55.332540 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:13:55.332553 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 20:13:55.333161 (+ 608us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:13:55.333210 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 20:13:55.333228 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:13:55.333353 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0923 20:13:55.333469 (+ 116us) server_negotiation.cc:299] Negotiation successful
0923 20:13:55.333592 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260923 20:13:55.334968 19774 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:13:55.382805 19925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:13:55.382201 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.129:43077 (local address 127.0.0.1:55696)
0923 20:13:55.382478 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0923 20:13:55.382589 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.170.129:43077: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260923 20:13:55.384016 19925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:13:55.383578 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.129:43077 (local address 127.0.0.1:55708)
0923 20:13:55.383748 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0923 20:13:55.383838 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.170.129:43077: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260923 20:13:55.442783 1706 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqUuku6/build/asan/bin/kudu with pid 19652
I20260923 20:13:55.444832 19925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:13:55.444230 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.129:43077 (local address 127.0.0.1:55716)
0923 20:13:55.444528 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0923 20:13:55.444646 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.170.129:43077: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
I20260923 20:13:55.446221 19925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:13:55.445699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.129:43077 (local address 127.0.0.1:55722)
0923 20:13:55.445975 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0923 20:13:55.446078 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.170.129:43077: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120}
I20260923 20:13:55.461983 1706 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskqUuku6/build/asan/bin/kudu pid 19782
I20260923 20:13:55.470454 19926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:13:55.462942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35550 (local address 127.1.170.131:46035)
0923 20:13:55.463123 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0923 20:13:55.463127 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 20:13:55.463194 (+ 67us) server_negotiation.cc:407] Connection header received
0923 20:13:55.463421 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:13:55.463426 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:13:55.463536 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:13:55.463629 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:13:55.464166 (+ 537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:13:55.465033 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:13:55.466969 (+ 1936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:13:55.467415 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:13:55.467530 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:13:55.468843 (+ 1313us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:13:55.468852 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:13:55.468867 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 20:13:55.469494 (+ 627us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:13:55.469547 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 20:13:55.469567 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:13:55.469661 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0923 20:13:55.469956 (+ 295us) server_negotiation.cc:299] Negotiation successful
0923 20:13:55.470056 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260923 20:13:55.471360 19904 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:13:55.527052 19925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:13:55.526421 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.129:43077 (local address 127.0.0.1:55724)
0923 20:13:55.526666 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0923 20:13:55.526833 (+ 167us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.170.129:43077: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
I20260923 20:13:55.528265 19925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:13:55.527818 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.129:43077 (local address 127.0.0.1:55726)
0923 20:13:55.528036 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0923 20:13:55.528124 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.170.129:43077: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111}
I20260923 20:13:55.597626 1706 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqUuku6/build/asan/bin/kudu with pid 19782
I20260923 20:13:55.617218 1706 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskqUuku6/build/asan/bin/kudu pid 19431
I20260923 20:13:55.618470 19492 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:13:55.731980 19925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:13:55.731174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.129:43077 (local address 127.0.0.1:55740)
0923 20:13:55.731578 (+ 404us) negotiation.cc:107] Waiting for socket to connect
0923 20:13:55.731696 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.170.129:43077: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":223}
I20260923 20:13:55.732136 1706 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqUuku6/build/asan/bin/kudu with pid 19431
2026-09-23T20:13:55Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (2837 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790194121577027-1706-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 20:13:55 dist-test-slave-th4g krb5kdc[19960](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 20:13:55 dist-test-slave-th4g krb5kdc[19960](info): set up 2 sockets
Sep 23 20:13:55 dist-test-slave-th4g krb5kdc[19960](info): commencing operation
krb5kdc: starting...
W20260923 20:13:56.890260 1706 mini_kdc.cc:121] Time spent starting KDC: real 1.115s user 0.000s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790194121577027-1706-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790194121577027-1706-0/krb5kdc/test-user.keytab.
Sep 23 20:13:57 dist-test-slave-th4g krb5kdc[19960](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194437, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T20:13:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T20:13:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.170.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.170.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.170.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790194121577027-1706-0/krb5kdc/kudu_127.1.170.190.keytab.
Entry for principal kudu/127.1.170.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790194121577027-1706-0/krb5kdc/kudu_127.1.170.190.keytab.
I20260923 20:13:57.147467 1706 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqUuku6/build/asan/bin/kudu
/tmp/dist-test-taskqUuku6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790194121577027-1706-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790194121577027-1706-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790194121577027-1706-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790194121577027-1706-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.170.190:43945
--webserver_interface=127.1.170.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790194121577027-1706-0/krb5kdc/kudu_127.1.170.190.keytab
--principal=kudu/127.1.170.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.170.148:40091
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.170.190:43945
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790194121577027-1706-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790194121577027-1706-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790194121577027-1706-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:13:57.473392 19973 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:13:57.473670 19973 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:13:57.473759 19973 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:13:57.480957 19973 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 20:13:57.481061 19973 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:13:57.481107 19973 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:13:57.481169 19973 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 20:13:57.481212 19973 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 20:13:57.490850 19973 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.170.148:40091
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790194121577027-1706-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790194121577027-1706-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.170.190:43945
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790194121577027-1706-0/krb5kdc/kudu_127.1.170.190.keytab
--principal=kudu/127.1.170.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.170.190:43945
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790194121577027-1706-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.170.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790194121577027-1706-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:13:57.492074 19973 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:13:57.493321 19973 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:13:57.504853 19978 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:13:57.504828 19979 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:13:57.504987 19981 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:13:57.505672 19973 server_base.cc:1034] running on GCE node
I20260923 20:13:57.506465 19973 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:13:57.507824 19973 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:13:57.509075 19973 hybrid_clock.cc:648] HybridClock initialized: now 1790194437509025 us; error 37 us; skew 500 ppm
I20260923 20:13:57.509426 19973 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:13:57 dist-test-slave-th4g krb5kdc[19960](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194437, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.170.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:13:57.512568 19973 init.cc:377] Logged in from keytab as kudu/127.1.170.190@KRBTEST.COM (short username kudu)
I20260923 20:13:57.514045 19973 webserver.cc:460] Webserver started at http://127.1.170.190:45917/ using document root <none> and password file <none>
I20260923 20:13:57.514672 19973 fs_manager.cc:359] Metadata directory not provided
I20260923 20:13:57.514779 19973 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:13:57.515015 19973 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:13:57.517848 19973 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790194121577027-1706-0/minicluster-data/master-0/data/instance:
uuid: "447a080aa83c4a28ad854bd8b9059b4d"
format_stamp: "Formatted at 2026-09-23 20:13:57 on dist-test-slave-th4g"
server_key: "0787bd354ba47a812ccfe7de882f44b1"
server_key_iv: "92c038cddd737695e3ebb05f1dfd03fb"
server_key_version: "encryptionkey@0"
I20260923 20:13:57.518610 19973 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790194121577027-1706-0/minicluster-data/master-0/wal/instance:
uuid: "447a080aa83c4a28ad854bd8b9059b4d"
format_stamp: "Formatted at 2026-09-23 20:13:57 on dist-test-slave-th4g"
server_key: "0787bd354ba47a812ccfe7de882f44b1"
server_key_iv: "92c038cddd737695e3ebb05f1dfd03fb"
server_key_version: "encryptionkey@0"
I20260923 20:13:57.523645 19973 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260923 20:13:57.527195 19988 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:13:57.528568 19973 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260923 20:13:57.528749 19973 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790194121577027-1706-0/minicluster-data/master-0/data,/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790194121577027-1706-0/minicluster-data/master-0/wal
uuid: "447a080aa83c4a28ad854bd8b9059b4d"
format_stamp: "Formatted at 2026-09-23 20:13:57 on dist-test-slave-th4g"
server_key: "0787bd354ba47a812ccfe7de882f44b1"
server_key_iv: "92c038cddd737695e3ebb05f1dfd03fb"
server_key_version: "encryptionkey@0"
I20260923 20:13:57.528929 19973 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790194121577027-1706-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790194121577027-1706-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790194121577027-1706-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:13:57.545687 19973 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:13:57.549813 19973 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:13:57.550153 19973 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:13:57.568876 19973 rpc_server.cc:307] RPC server started. Bound to: 127.1.170.190:43945
I20260923 20:13:57.568974 20039 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.170.190:43945 every 8 connection(s)
I20260923 20:13:57.570268 19973 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790194121577027-1706-0/minicluster-data/master-0/data/info.pb
I20260923 20:13:57.573963 20040 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:13:57.580530 1706 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqUuku6/build/asan/bin/kudu as pid 19973
I20260923 20:13:57.580732 1706 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790194121577027-1706-0/minicluster-data/master-0/wal/instance
I20260923 20:13:57.581274 1706 external_mini_cluster.cc:1428] Setting key 2dad971f618e50ab06e5cdf4a2056e9b
I20260923 20:13:57.584826 20040 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 447a080aa83c4a28ad854bd8b9059b4d: Bootstrap starting.
I20260923 20:13:57.588559 20040 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 447a080aa83c4a28ad854bd8b9059b4d: Neither blocks nor log segments found. Creating new log.
I20260923 20:13:57.589630 20040 log.cc:824] T 00000000000000000000000000000000 P 447a080aa83c4a28ad854bd8b9059b4d: Log is configured to *not* fsync() on all Append() calls
I20260923 20:13:57.592624 20040 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 447a080aa83c4a28ad854bd8b9059b4d: No bootstrap required, opened a new log
Sep 23 20:13:57 dist-test-slave-th4g krb5kdc[19960](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194437, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.170.190@KRBTEST.COM
I20260923 20:13:57.601384 20040 raft_consensus.cc:348] T 00000000000000000000000000000000 P 447a080aa83c4a28ad854bd8b9059b4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "447a080aa83c4a28ad854bd8b9059b4d" member_type: VOTER last_known_addr { host: "127.1.170.190" port: 43945 } }
I20260923 20:13:57.601802 20040 raft_consensus.cc:374] T 00000000000000000000000000000000 P 447a080aa83c4a28ad854bd8b9059b4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:13:57.601946 20040 raft_consensus.cc:729] T 00000000000000000000000000000000 P 447a080aa83c4a28ad854bd8b9059b4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 447a080aa83c4a28ad854bd8b9059b4d, State: Initialized, Role: FOLLOWER
I20260923 20:13:57.602638 20040 consensus_queue.cc:260] T 00000000000000000000000000000000 P 447a080aa83c4a28ad854bd8b9059b4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "447a080aa83c4a28ad854bd8b9059b4d" member_type: VOTER last_known_addr { host: "127.1.170.190" port: 43945 } }
I20260923 20:13:57.602973 20040 raft_consensus.cc:388] T 00000000000000000000000000000000 P 447a080aa83c4a28ad854bd8b9059b4d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 20:13:57.603118 20040 raft_consensus.cc:482] T 00000000000000000000000000000000 P 447a080aa83c4a28ad854bd8b9059b4d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 20:13:57.603332 20040 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 447a080aa83c4a28ad854bd8b9059b4d [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:13:57.605374 20043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:13:57.583173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35330 (local address 127.1.170.190:43945)
0923 20:13:57.583940 (+ 767us) server_negotiation.cc:206] Beginning negotiation
0923 20:13:57.583951 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0923 20:13:57.583995 (+ 44us) server_negotiation.cc:407] Connection header received
0923 20:13:57.585726 (+ 1731us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:13:57.585747 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:13:57.586182 (+ 435us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:13:57.586501 (+ 319us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:13:57.588410 (+ 1909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:13:57.590012 (+ 1602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:13:57.591715 (+ 1703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:13:57.592370 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:13:57.592498 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:13:57.597279 (+ 4781us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:13:57.597287 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:13:57.597301 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:13:57.597351 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:13:57.600519 (+ 3168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:13:57.601321 (+ 802us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:13:57.601327 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:13:57.601333 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:13:57.601511 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:13:57.601990 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:13:57.601998 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:13:57.602001 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:13:57.602703 (+ 702us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:13:57.603046 (+ 343us) server_negotiation.cc:1035] Waiting for connection context
0923 20:13:57.603489 (+ 443us) server_negotiation.cc:299] Negotiation successful
0923 20:13:57.603966 (+ 477us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":578,"thread_start_us":385,"threads_started":1}
I20260923 20:13:57.606807 20040 raft_consensus.cc:504] T 00000000000000000000000000000000 P 447a080aa83c4a28ad854bd8b9059b4d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "447a080aa83c4a28ad854bd8b9059b4d" member_type: VOTER last_known_addr { host: "127.1.170.190" port: 43945 } }
I20260923 20:13:57.607348 20040 leader_election.cc:302] T 00000000000000000000000000000000 P 447a080aa83c4a28ad854bd8b9059b4d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 447a080aa83c4a28ad854bd8b9059b4d; no voters:
I20260923 20:13:57.607990 20040 leader_election.cc:288] T 00000000000000000000000000000000 P 447a080aa83c4a28ad854bd8b9059b4d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 20:13:57.608187 20045 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 447a080aa83c4a28ad854bd8b9059b4d [term 1 FOLLOWER]: Leader election won for term 1
I20260923 20:13:57.609212 20045 raft_consensus.cc:686] T 00000000000000000000000000000000 P 447a080aa83c4a28ad854bd8b9059b4d [term 1 LEADER]: Becoming Leader. State: Replica: 447a080aa83c4a28ad854bd8b9059b4d, State: Running, Role: LEADER
I20260923 20:13:57.609736 20045 consensus_queue.cc:237] T 00000000000000000000000000000000 P 447a080aa83c4a28ad854bd8b9059b4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "447a080aa83c4a28ad854bd8b9059b4d" member_type: VOTER last_known_addr { host: "127.1.170.190" port: 43945 } }
I20260923 20:13:57.610270 20040 sys_catalog.cc:564] T 00000000000000000000000000000000 P 447a080aa83c4a28ad854bd8b9059b4d [sys.catalog]: configured and running, proceeding with master startup.
I20260923 20:13:57.613221 20046 sys_catalog.cc:455] T 00000000000000000000000000000000 P 447a080aa83c4a28ad854bd8b9059b4d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "447a080aa83c4a28ad854bd8b9059b4d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "447a080aa83c4a28ad854bd8b9059b4d" member_type: VOTER last_known_addr { host: "127.1.170.190" port: 43945 } } }
I20260923 20:13:57.614068 20046 sys_catalog.cc:458] T 00000000000000000000000000000000 P 447a080aa83c4a28ad854bd8b9059b4d [sys.catalog]: This master's current role is: LEADER
I20260923 20:13:57.613226 20047 sys_catalog.cc:455] T 00000000000000000000000000000000 P 447a080aa83c4a28ad854bd8b9059b4d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 447a080aa83c4a28ad854bd8b9059b4d. Latest consensus state: current_term: 1 leader_uuid: "447a080aa83c4a28ad854bd8b9059b4d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "447a080aa83c4a28ad854bd8b9059b4d" member_type: VOTER last_known_addr { host: "127.1.170.190" port: 43945 } } }
I20260923 20:13:57.614672 20047 sys_catalog.cc:458] T 00000000000000000000000000000000 P 447a080aa83c4a28ad854bd8b9059b4d [sys.catalog]: This master's current role is: LEADER
I20260923 20:13:57.616021 20054 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 20:13:57.620455 20054 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 20:13:57.628417 20054 catalog_manager.cc:1347] Generated new cluster ID: cc4d55f8c458409ca8f677e11ff34235
I20260923 20:13:57.628535 20054 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 20:13:57.645504 20054 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 20:13:57.647338 20054 catalog_manager.cc:1504] Loading token signing keys...
I20260923 20:13:57.658820 20054 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 447a080aa83c4a28ad854bd8b9059b4d: Generated new TSK 0
I20260923 20:13:57.659914 20054 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.170.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.170.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.170.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790194121577027-1706-0/krb5kdc/kudu_127.1.170.129.keytab.
Entry for principal kudu/127.1.170.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790194121577027-1706-0/krb5kdc/kudu_127.1.170.129.keytab.
I20260923 20:13:57.721766 1706 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqUuku6/build/asan/bin/kudu
/tmp/dist-test-taskqUuku6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790194121577027-1706-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790194121577027-1706-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790194121577027-1706-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790194121577027-1706-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.170.129:0
--local_ip_for_outbound_sockets=127.1.170.129
--webserver_interface=127.1.170.129
--webserver_port=0
--tserver_master_addrs=127.1.170.190:43945
--keytab_file=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790194121577027-1706-0/krb5kdc/kudu_127.1.170.129.keytab
--principal=kudu/127.1.170.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.170.148:40091
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790194121577027-1706-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790194121577027-1706-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790194121577027-1706-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:13:58.055157 20066 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:13:58.055461 20066 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:13:58.055554 20066 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:13:58.062603 20066 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:13:58.062711 20066 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:13:58.062819 20066 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.170.129
I20260923 20:13:58.072400 20066 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.170.148:40091
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790194121577027-1706-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790194121577027-1706-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790194121577027-1706-0/krb5kdc/kudu_127.1.170.129.keytab
--principal=kudu/127.1.170.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.170.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790194121577027-1706-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.170.129
--webserver_port=0
--tserver_master_addrs=127.1.170.190:43945
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.170.129
--log_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790194121577027-1706-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:13:58.073621 20066 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:13:58.074978 20066 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:13:58.088013 20072 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:13:58.088137 20074 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:13:58.088063 20071 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:13:58.089408 20066 server_base.cc:1034] running on GCE node
I20260923 20:13:58.090140 20066 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:13:58.091133 20066 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:13:58.092381 20066 hybrid_clock.cc:648] HybridClock initialized: now 1790194438092313 us; error 49 us; skew 500 ppm
I20260923 20:13:58.092736 20066 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:13:58 dist-test-slave-th4g krb5kdc[19960](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194438, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.170.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:13:58.095878 20066 init.cc:377] Logged in from keytab as kudu/127.1.170.129@KRBTEST.COM (short username kudu)
I20260923 20:13:58.097362 20066 webserver.cc:460] Webserver started at http://127.1.170.129:46793/ using document root <none> and password file <none>
I20260923 20:13:58.098095 20066 fs_manager.cc:359] Metadata directory not provided
I20260923 20:13:58.098229 20066 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:13:58.098541 20066 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:13:58.101372 20066 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790194121577027-1706-0/minicluster-data/ts-0/data/instance:
uuid: "9de5527c801840c8ba994331b1e36b4e"
format_stamp: "Formatted at 2026-09-23 20:13:58 on dist-test-slave-th4g"
server_key: "187adb4a72f4aa884ebb25028c20b32f"
server_key_iv: "944337fbdf34fc226b4426f7d893c155"
server_key_version: "encryptionkey@0"
I20260923 20:13:58.102234 20066 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790194121577027-1706-0/minicluster-data/ts-0/wal/instance:
uuid: "9de5527c801840c8ba994331b1e36b4e"
format_stamp: "Formatted at 2026-09-23 20:13:58 on dist-test-slave-th4g"
server_key: "187adb4a72f4aa884ebb25028c20b32f"
server_key_iv: "944337fbdf34fc226b4426f7d893c155"
server_key_version: "encryptionkey@0"
I20260923 20:13:58.107362 20066 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260923 20:13:58.110858 20081 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:13:58.112323 20066 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.003s
I20260923 20:13:58.112587 20066 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790194121577027-1706-0/minicluster-data/ts-0/data,/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790194121577027-1706-0/minicluster-data/ts-0/wal
uuid: "9de5527c801840c8ba994331b1e36b4e"
format_stamp: "Formatted at 2026-09-23 20:13:58 on dist-test-slave-th4g"
server_key: "187adb4a72f4aa884ebb25028c20b32f"
server_key_iv: "944337fbdf34fc226b4426f7d893c155"
server_key_version: "encryptionkey@0"
I20260923 20:13:58.112823 20066 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790194121577027-1706-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790194121577027-1706-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790194121577027-1706-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:13:58.134428 20066 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:13:58.138345 20066 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:13:58.138638 20066 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:13:58.139573 20066 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 20:13:58.140829 20066 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 20:13:58.140931 20066 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 20:13:58.141075 20066 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 20:13:58.141144 20066 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 20:13:58.174078 20066 rpc_server.cc:307] RPC server started. Bound to: 127.1.170.129:43531
I20260923 20:13:58.174154 20193 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.170.129:43531 every 8 connection(s)
I20260923 20:13:58.175304 20066 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790194121577027-1706-0/minicluster-data/ts-0/data/info.pb
I20260923 20:13:58.175571 1706 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqUuku6/build/asan/bin/kudu as pid 20066
I20260923 20:13:58.175737 1706 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790194121577027-1706-0/minicluster-data/ts-0/wal/instance
I20260923 20:13:58.176280 1706 external_mini_cluster.cc:1428] Setting key 3250f16058de80a264910f28a60a9905
Sep 23 20:13:58 dist-test-slave-th4g krb5kdc[19960](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194438, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.170.129@KRBTEST.COM for kudu/127.1.170.190@KRBTEST.COM
I20260923 20:13:58.199033 20196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:13:58.178068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.170.129:41337 (local address 127.1.170.190:43945)
0923 20:13:58.178538 (+ 470us) server_negotiation.cc:206] Beginning negotiation
0923 20:13:58.178542 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 20:13:58.179744 (+ 1202us) server_negotiation.cc:407] Connection header received
0923 20:13:58.182685 (+ 2941us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:13:58.182689 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:13:58.182755 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:13:58.182847 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:13:58.184847 (+ 2000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:13:58.185598 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:13:58.188046 (+ 2448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:13:58.188424 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:13:58.188540 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:13:58.192102 (+ 3562us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:13:58.192111 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:13:58.192119 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:13:58.192165 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:13:58.195708 (+ 3543us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:13:58.196422 (+ 714us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:13:58.196431 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:13:58.196437 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:13:58.196586 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:13:58.197015 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:13:58.197023 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:13:58.197029 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:13:58.197400 (+ 371us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:13:58.197549 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0923 20:13:58.198294 (+ 745us) server_negotiation.cc:299] Negotiation successful
0923 20:13:58.198553 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":368,"thread_start_us":262,"threads_started":1}
I20260923 20:13:58.199604 20197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:13:58.178730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.190:43945 (local address 127.1.170.129:41337)
0923 20:13:58.179562 (+ 832us) negotiation.cc:107] Waiting for socket to connect
0923 20:13:58.179621 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0923 20:13:58.182379 (+ 2758us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:13:58.183099 (+ 720us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:13:58.183113 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:13:58.183875 (+ 762us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:13:58.184661 (+ 786us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:13:58.184686 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:13:58.185748 (+ 1062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:13:58.185756 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:13:58.187906 (+ 2150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:13:58.187922 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:13:58.188625 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:13:58.188629 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:13:58.188814 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:13:58.189509 (+ 695us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:13:58.189534 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:13:58.191858 (+ 2324us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:13:58.195852 (+ 3994us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:13:58.195858 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:13:58.195870 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:13:58.196285 (+ 415us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:13:58.196690 (+ 405us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:13:58.196694 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:13:58.196696 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:13:58.196878 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:13:58.197524 (+ 646us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:13:58.197530 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:13:58.197825 (+ 295us) client_negotiation.cc:769] Sending connection context
0923 20:13:58.198134 (+ 309us) client_negotiation.cc:240] Negotiation successful
0923 20:13:58.198512 (+ 378us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":527,"thread_start_us":322,"threads_started":1}
I20260923 20:13:58.201208 20194 heartbeater.cc:347] Connected to a master server at 127.1.170.190:43945
I20260923 20:13:58.201507 20194 heartbeater.cc:464] Registering TS with master...
I20260923 20:13:58.202229 20194 heartbeater.cc:511] Master 127.1.170.190:43945 requested a full tablet report, sending...
I20260923 20:13:58.204221 20005 ts_manager.cc:194] Registered new tserver with Master: 9de5527c801840c8ba994331b1e36b4e (127.1.170.129:43531)
WARNING: no policy specified for kudu/127.1.170.130@KRBTEST.COM; defaulting to no policy
I20260923 20:13:58.205677 20005 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.170.129@KRBTEST.COM'} at 127.1.170.129:41337
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.170.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.170.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790194121577027-1706-0/krb5kdc/kudu_127.1.170.130.keytab.
Entry for principal kudu/127.1.170.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790194121577027-1706-0/krb5kdc/kudu_127.1.170.130.keytab.
I20260923 20:13:58.237329 1706 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqUuku6/build/asan/bin/kudu
/tmp/dist-test-taskqUuku6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790194121577027-1706-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790194121577027-1706-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790194121577027-1706-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790194121577027-1706-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.170.130:0
--local_ip_for_outbound_sockets=127.1.170.130
--webserver_interface=127.1.170.130
--webserver_port=0
--tserver_master_addrs=127.1.170.190:43945
--keytab_file=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790194121577027-1706-0/krb5kdc/kudu_127.1.170.130.keytab
--principal=kudu/127.1.170.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.170.148:40091
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790194121577027-1706-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790194121577027-1706-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790194121577027-1706-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:13:58.559266 20200 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:13:58.559546 20200 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:13:58.559639 20200 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:13:58.566706 20200 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:13:58.566824 20200 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:13:58.566936 20200 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.170.130
I20260923 20:13:58.576382 20200 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.170.148:40091
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790194121577027-1706-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790194121577027-1706-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790194121577027-1706-0/krb5kdc/kudu_127.1.170.130.keytab
--principal=kudu/127.1.170.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.170.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790194121577027-1706-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.170.130
--webserver_port=0
--tserver_master_addrs=127.1.170.190:43945
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.170.130
--log_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790194121577027-1706-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:13:58.577550 20200 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:13:58.578794 20200 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:13:58.592170 20205 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:13:58.592123 20206 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:13:58.592118 20208 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:13:58.592945 20200 server_base.cc:1034] running on GCE node
I20260923 20:13:58.593540 20200 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:13:58.594607 20200 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:13:58.595829 20200 hybrid_clock.cc:648] HybridClock initialized: now 1790194438595781 us; error 34 us; skew 500 ppm
I20260923 20:13:58.596133 20200 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:13:58 dist-test-slave-th4g krb5kdc[19960](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194438, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.170.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:13:58.599254 20200 init.cc:377] Logged in from keytab as kudu/127.1.170.130@KRBTEST.COM (short username kudu)
I20260923 20:13:58.600745 20200 webserver.cc:460] Webserver started at http://127.1.170.130:38595/ using document root <none> and password file <none>
I20260923 20:13:58.601421 20200 fs_manager.cc:359] Metadata directory not provided
I20260923 20:13:58.601572 20200 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:13:58.601935 20200 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:13:58.604897 20200 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790194121577027-1706-0/minicluster-data/ts-1/data/instance:
uuid: "5b602cdfb97d403191baafbca741cbcf"
format_stamp: "Formatted at 2026-09-23 20:13:58 on dist-test-slave-th4g"
server_key: "5b4d339dd7e9e65a7f98f47c3bf0feae"
server_key_iv: "2ebf8682840f38e15ee24323f46db491"
server_key_version: "encryptionkey@0"
I20260923 20:13:58.605705 20200 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790194121577027-1706-0/minicluster-data/ts-1/wal/instance:
uuid: "5b602cdfb97d403191baafbca741cbcf"
format_stamp: "Formatted at 2026-09-23 20:13:58 on dist-test-slave-th4g"
server_key: "5b4d339dd7e9e65a7f98f47c3bf0feae"
server_key_iv: "2ebf8682840f38e15ee24323f46db491"
server_key_version: "encryptionkey@0"
I20260923 20:13:58.610945 20200 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260923 20:13:58.614384 20215 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:13:58.615630 20200 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260923 20:13:58.615799 20200 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790194121577027-1706-0/minicluster-data/ts-1/data,/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790194121577027-1706-0/minicluster-data/ts-1/wal
uuid: "5b602cdfb97d403191baafbca741cbcf"
format_stamp: "Formatted at 2026-09-23 20:13:58 on dist-test-slave-th4g"
server_key: "5b4d339dd7e9e65a7f98f47c3bf0feae"
server_key_iv: "2ebf8682840f38e15ee24323f46db491"
server_key_version: "encryptionkey@0"
I20260923 20:13:58.615996 20200 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790194121577027-1706-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790194121577027-1706-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790194121577027-1706-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:13:58.646922 20200 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:13:58.650964 20200 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:13:58.651269 20200 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:13:58.652213 20200 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 20:13:58.653551 20200 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 20:13:58.653650 20200 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 20:13:58.653797 20200 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 20:13:58.653867 20200 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 20:13:58.686511 20200 rpc_server.cc:307] RPC server started. Bound to: 127.1.170.130:40877
I20260923 20:13:58.686591 20327 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.170.130:40877 every 8 connection(s)
I20260923 20:13:58.687770 20200 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790194121577027-1706-0/minicluster-data/ts-1/data/info.pb
I20260923 20:13:58.691020 1706 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqUuku6/build/asan/bin/kudu as pid 20200
I20260923 20:13:58.691215 1706 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790194121577027-1706-0/minicluster-data/ts-1/wal/instance
I20260923 20:13:58.691766 1706 external_mini_cluster.cc:1428] Setting key 716719b7fdc3cc7055b2de5611dad484
Sep 23 20:13:58 dist-test-slave-th4g krb5kdc[19960](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194438, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.170.130@KRBTEST.COM for kudu/127.1.170.190@KRBTEST.COM
I20260923 20:13:58.707850 20196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:13:58.690222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.170.130:55185 (local address 127.1.170.190:43945)
0923 20:13:58.690407 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0923 20:13:58.690412 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 20:13:58.691605 (+ 1193us) server_negotiation.cc:407] Connection header received
0923 20:13:58.693550 (+ 1945us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:13:58.693558 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:13:58.693629 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:13:58.693706 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:13:58.695218 (+ 1512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:13:58.696449 (+ 1231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:13:58.697959 (+ 1510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:13:58.698396 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:13:58.698527 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:13:58.702093 (+ 3566us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:13:58.702098 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:13:58.702101 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:13:58.702132 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:13:58.704767 (+ 2635us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:13:58.705541 (+ 774us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:13:58.705545 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:13:58.705547 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:13:58.705670 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:13:58.706159 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:13:58.706163 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:13:58.706165 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:13:58.706416 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:13:58.706582 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0923 20:13:58.707264 (+ 682us) server_negotiation.cc:299] Negotiation successful
0923 20:13:58.707506 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260923 20:13:58.708757 20330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:13:58.690612 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.190:43945 (local address 127.1.170.130:55185)
0923 20:13:58.691416 (+ 804us) negotiation.cc:107] Waiting for socket to connect
0923 20:13:58.691472 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0923 20:13:58.693295 (+ 1823us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:13:58.693992 (+ 697us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:13:58.693999 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:13:58.694479 (+ 480us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:13:58.695017 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:13:58.695036 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:13:58.696595 (+ 1559us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:13:58.696600 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:13:58.697792 (+ 1192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:13:58.697804 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:13:58.698617 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:13:58.698624 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:13:58.698845 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:13:58.699507 (+ 662us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:13:58.699529 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:13:58.701838 (+ 2309us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:13:58.704936 (+ 3098us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:13:58.704946 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:13:58.704989 (+ 43us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:13:58.705387 (+ 398us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:13:58.705780 (+ 393us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:13:58.705785 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:13:58.705789 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:13:58.706037 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:13:58.706549 (+ 512us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:13:58.706558 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:13:58.706897 (+ 339us) client_negotiation.cc:769] Sending connection context
0923 20:13:58.707163 (+ 266us) client_negotiation.cc:240] Negotiation successful
0923 20:13:58.707660 (+ 497us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":527,"thread_start_us":331,"threads_started":1}
I20260923 20:13:58.710362 20328 heartbeater.cc:347] Connected to a master server at 127.1.170.190:43945
I20260923 20:13:58.710678 20328 heartbeater.cc:464] Registering TS with master...
I20260923 20:13:58.711488 20328 heartbeater.cc:511] Master 127.1.170.190:43945 requested a full tablet report, sending...
I20260923 20:13:58.712965 20005 ts_manager.cc:194] Registered new tserver with Master: 5b602cdfb97d403191baafbca741cbcf (127.1.170.130:40877)
I20260923 20:13:58.713778 20005 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.170.130@KRBTEST.COM'} at 127.1.170.130:55185
WARNING: no policy specified for kudu/127.1.170.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.170.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.170.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790194121577027-1706-0/krb5kdc/kudu_127.1.170.131.keytab.
Entry for principal kudu/127.1.170.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790194121577027-1706-0/krb5kdc/kudu_127.1.170.131.keytab.
I20260923 20:13:58.749034 1706 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqUuku6/build/asan/bin/kudu
/tmp/dist-test-taskqUuku6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790194121577027-1706-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790194121577027-1706-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790194121577027-1706-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790194121577027-1706-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.170.131:0
--local_ip_for_outbound_sockets=127.1.170.131
--webserver_interface=127.1.170.131
--webserver_port=0
--tserver_master_addrs=127.1.170.190:43945
--keytab_file=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790194121577027-1706-0/krb5kdc/kudu_127.1.170.131.keytab
--principal=kudu/127.1.170.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.170.148:40091
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790194121577027-1706-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790194121577027-1706-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790194121577027-1706-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:13:59.075704 20333 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:13:59.076033 20333 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:13:59.076211 20333 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:13:59.083536 20333 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:13:59.083703 20333 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:13:59.083910 20333 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.170.131
I20260923 20:13:59.093863 20333 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.170.148:40091
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790194121577027-1706-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790194121577027-1706-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790194121577027-1706-0/krb5kdc/kudu_127.1.170.131.keytab
--principal=kudu/127.1.170.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.170.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790194121577027-1706-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.170.131
--webserver_port=0
--tserver_master_addrs=127.1.170.190:43945
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.170.131
--log_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790194121577027-1706-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:13:59.095242 20333 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:13:59.096540 20333 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:13:59.109805 20338 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:13:59.110069 20339 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:13:59.110184 20341 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:13:59.111430 20333 server_base.cc:1034] running on GCE node
I20260923 20:13:59.112051 20333 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:13:59.113029 20333 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:13:59.114286 20333 hybrid_clock.cc:648] HybridClock initialized: now 1790194439114219 us; error 39 us; skew 500 ppm
I20260923 20:13:59.114596 20333 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:13:59 dist-test-slave-th4g krb5kdc[19960](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194439, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.170.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:13:59.117838 20333 init.cc:377] Logged in from keytab as kudu/127.1.170.131@KRBTEST.COM (short username kudu)
I20260923 20:13:59.119418 20333 webserver.cc:460] Webserver started at http://127.1.170.131:45633/ using document root <none> and password file <none>
I20260923 20:13:59.120132 20333 fs_manager.cc:359] Metadata directory not provided
I20260923 20:13:59.120276 20333 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:13:59.120574 20333 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:13:59.123482 20333 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790194121577027-1706-0/minicluster-data/ts-2/data/instance:
uuid: "49f35fde25484d6483b1d067e92ceeac"
format_stamp: "Formatted at 2026-09-23 20:13:59 on dist-test-slave-th4g"
server_key: "7ee9a5b06656510919b49dec05f936aa"
server_key_iv: "d839f8be3573848332b8b5e7d12a331c"
server_key_version: "encryptionkey@0"
I20260923 20:13:59.124245 20333 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790194121577027-1706-0/minicluster-data/ts-2/wal/instance:
uuid: "49f35fde25484d6483b1d067e92ceeac"
format_stamp: "Formatted at 2026-09-23 20:13:59 on dist-test-slave-th4g"
server_key: "7ee9a5b06656510919b49dec05f936aa"
server_key_iv: "d839f8be3573848332b8b5e7d12a331c"
server_key_version: "encryptionkey@0"
I20260923 20:13:59.129228 20333 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260923 20:13:59.132987 20348 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:13:59.134434 20333 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 20:13:59.134603 20333 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790194121577027-1706-0/minicluster-data/ts-2/data,/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790194121577027-1706-0/minicluster-data/ts-2/wal
uuid: "49f35fde25484d6483b1d067e92ceeac"
format_stamp: "Formatted at 2026-09-23 20:13:59 on dist-test-slave-th4g"
server_key: "7ee9a5b06656510919b49dec05f936aa"
server_key_iv: "d839f8be3573848332b8b5e7d12a331c"
server_key_version: "encryptionkey@0"
I20260923 20:13:59.134805 20333 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790194121577027-1706-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790194121577027-1706-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790194121577027-1706-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:13:59.149416 20333 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:13:59.153415 20333 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:13:59.153705 20333 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:13:59.154713 20333 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 20:13:59.156316 20333 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 20:13:59.156430 20333 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 20:13:59.156574 20333 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 20:13:59.156627 20333 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 20:13:59.190964 20333 rpc_server.cc:307] RPC server started. Bound to: 127.1.170.131:37953
I20260923 20:13:59.191092 20460 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.170.131:37953 every 8 connection(s)
I20260923 20:13:59.192266 20333 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790194121577027-1706-0/minicluster-data/ts-2/data/info.pb
I20260923 20:13:59.193075 1706 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqUuku6/build/asan/bin/kudu as pid 20333
I20260923 20:13:59.193269 1706 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790194121577027-1706-0/minicluster-data/ts-2/wal/instance
I20260923 20:13:59.193856 1706 external_mini_cluster.cc:1428] Setting key 54c38f9a4c7c7b23339eb7c62fd31c80
Sep 23 20:13:59 dist-test-slave-th4g krb5kdc[19960](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194439, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.170.131@KRBTEST.COM for kudu/127.1.170.190@KRBTEST.COM
I20260923 20:13:59.208628 20194 heartbeater.cc:503] Master 127.1.170.190:43945 was elected leader, sending a full tablet report...
I20260923 20:13:59.214689 20196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:13:59.195092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.170.131:50397 (local address 127.1.170.190:43945)
0923 20:13:59.195273 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0923 20:13:59.195277 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 20:13:59.196849 (+ 1572us) server_negotiation.cc:407] Connection header received
0923 20:13:59.198660 (+ 1811us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:13:59.198664 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:13:59.198737 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:13:59.198827 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:13:59.200376 (+ 1549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:13:59.201187 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:13:59.202892 (+ 1705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:13:59.203328 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:13:59.203440 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:13:59.207215 (+ 3775us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:13:59.207220 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:13:59.207223 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:13:59.207259 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:13:59.211186 (+ 3927us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:13:59.212144 (+ 958us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:13:59.212149 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:13:59.212153 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:13:59.212297 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:13:59.212767 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:13:59.212772 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:13:59.212775 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:13:59.213056 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:13:59.213217 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0923 20:13:59.214054 (+ 837us) server_negotiation.cc:299] Negotiation successful
0923 20:13:59.214283 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260923 20:13:59.215637 20463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:13:59.195714 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.190:43945 (local address 127.1.170.131:50397)
0923 20:13:59.196533 (+ 819us) negotiation.cc:107] Waiting for socket to connect
0923 20:13:59.196585 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0923 20:13:59.198346 (+ 1761us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:13:59.199079 (+ 733us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:13:59.199087 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:13:59.199633 (+ 546us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:13:59.200191 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:13:59.200208 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:13:59.201358 (+ 1150us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:13:59.201362 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:13:59.202711 (+ 1349us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:13:59.202723 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:13:59.203553 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:13:59.203559 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:13:59.203801 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:13:59.204478 (+ 677us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:13:59.204503 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:13:59.206985 (+ 2482us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:13:59.211466 (+ 4481us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:13:59.211477 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:13:59.211491 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:13:59.212004 (+ 513us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:13:59.212414 (+ 410us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:13:59.212419 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:13:59.212423 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:13:59.212636 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:13:59.213188 (+ 552us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:13:59.213197 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:13:59.213649 (+ 452us) client_negotiation.cc:769] Sending connection context
0923 20:13:59.213952 (+ 303us) client_negotiation.cc:240] Negotiation successful
0923 20:13:59.214517 (+ 565us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":501,"thread_start_us":278,"threads_started":1}
I20260923 20:13:59.217118 20461 heartbeater.cc:347] Connected to a master server at 127.1.170.190:43945
I20260923 20:13:59.217437 20461 heartbeater.cc:464] Registering TS with master...
I20260923 20:13:59.218252 20461 heartbeater.cc:511] Master 127.1.170.190:43945 requested a full tablet report, sending...
I20260923 20:13:59.219697 20004 ts_manager.cc:194] Registered new tserver with Master: 49f35fde25484d6483b1d067e92ceeac (127.1.170.131:37953)
I20260923 20:13:59.220887 20004 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.170.131@KRBTEST.COM'} at 127.1.170.131:50397
I20260923 20:13:59.222391 1706 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260923 20:13:59.244022 20196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:13:59.227289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35332 (local address 127.1.170.190:43945)
0923 20:13:59.227480 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0923 20:13:59.227485 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 20:13:59.227914 (+ 429us) server_negotiation.cc:407] Connection header received
0923 20:13:59.228035 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:13:59.228039 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:13:59.228104 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:13:59.228201 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:13:59.230215 (+ 2014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:13:59.230988 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:13:59.232763 (+ 1775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:13:59.233269 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:13:59.233400 (+ 131us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:13:59.236794 (+ 3394us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:13:59.236798 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:13:59.236802 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:13:59.236838 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:13:59.240327 (+ 3489us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:13:59.241390 (+ 1063us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:13:59.241414 (+ 24us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:13:59.241416 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:13:59.241600 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:13:59.242172 (+ 572us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:13:59.242176 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:13:59.242178 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:13:59.242548 (+ 370us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:13:59.242805 (+ 257us) server_negotiation.cc:1035] Waiting for connection context
0923 20:13:59.243557 (+ 752us) server_negotiation.cc:299] Negotiation successful
0923 20:13:59.243752 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260923 20:13:59.249197 20004 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35332:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 20:13:59.257275 20004 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 20:13:59.292593 20475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:13:59.270531 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54200 (local address 127.1.170.130:40877)
0923 20:13:59.283775 (+ 13244us) server_negotiation.cc:206] Beginning negotiation
0923 20:13:59.283787 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0923 20:13:59.283820 (+ 33us) server_negotiation.cc:407] Connection header received
0923 20:13:59.283947 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:13:59.283954 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:13:59.284206 (+ 252us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:13:59.284428 (+ 222us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:13:59.285003 (+ 575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:13:59.286721 (+ 1718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:13:59.289718 (+ 2997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:13:59.291340 (+ 1622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:13:59.291480 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:13:59.291661 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0923 20:13:59.292076 (+ 415us) server_negotiation.cc:299] Negotiation successful
0923 20:13:59.292240 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13039,"thread_start_us":13472,"threads_started":1}
I20260923 20:13:59.293022 20473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:13:59.270454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.129:43531 (local address 127.0.0.1:49718)
0923 20:13:59.283644 (+ 13190us) negotiation.cc:107] Waiting for socket to connect
0923 20:13:59.283664 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0923 20:13:59.283882 (+ 218us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:13:59.285079 (+ 1197us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:13:59.285085 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:13:59.285116 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:13:59.285528 (+ 412us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:13:59.285541 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:13:59.287467 (+ 1926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:13:59.287471 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:13:59.289672 (+ 2201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:13:59.289684 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:13:59.291897 (+ 2213us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:13:59.291902 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:13:59.292115 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:13:59.292144 (+ 29us) client_negotiation.cc:769] Sending connection context
0923 20:13:59.292229 (+ 85us) client_negotiation.cc:240] Negotiation successful
0923 20:13:59.292363 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13047,"thread_start_us":11767,"threads_started":1}
I20260923 20:13:59.293211 20474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:13:59.270274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.130:40877 (local address 127.0.0.1:54200)
0923 20:13:59.282781 (+ 12507us) negotiation.cc:107] Waiting for socket to connect
0923 20:13:59.282849 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0923 20:13:59.283111 (+ 262us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:13:59.284436 (+ 1325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:13:59.284443 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:13:59.284472 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:13:59.284843 (+ 371us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:13:59.284853 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:13:59.286907 (+ 2054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:13:59.286913 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:13:59.289515 (+ 2602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:13:59.289532 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:13:59.291554 (+ 2022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:13:59.291560 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:13:59.291764 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:13:59.291838 (+ 74us) client_negotiation.cc:769] Sending connection context
0923 20:13:59.292030 (+ 192us) client_negotiation.cc:240] Negotiation successful
0923 20:13:59.292230 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12140,"thread_start_us":13088,"threads_started":1}
I20260923 20:13:59.293522 20476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:13:59.270868 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49718 (local address 127.1.170.129:43531)
0923 20:13:59.281729 (+ 10861us) server_negotiation.cc:206] Beginning negotiation
0923 20:13:59.281743 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0923 20:13:59.283768 (+ 2025us) server_negotiation.cc:407] Connection header received
0923 20:13:59.284468 (+ 700us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:13:59.284480 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:13:59.284740 (+ 260us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:13:59.284968 (+ 228us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:13:59.285739 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:13:59.287281 (+ 1542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:13:59.289840 (+ 2559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:13:59.291340 (+ 1500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:13:59.291464 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:13:59.291696 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0923 20:13:59.292988 (+ 1292us) server_negotiation.cc:299] Negotiation successful
0923 20:13:59.293167 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10577,"thread_start_us":10893,"threads_started":1}
I20260923 20:13:59.296936 20478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:13:59.285453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.131:37953 (local address 127.0.0.1:43050)
0923 20:13:59.289059 (+ 3606us) negotiation.cc:107] Waiting for socket to connect
0923 20:13:59.289080 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0923 20:13:59.289224 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:13:59.289946 (+ 722us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:13:59.289952 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:13:59.289979 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:13:59.290318 (+ 339us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:13:59.290325 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:13:59.291886 (+ 1561us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:13:59.291891 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:13:59.294453 (+ 2562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:13:59.294466 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:13:59.296164 (+ 1698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:13:59.296172 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:13:59.296337 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:13:59.296369 (+ 32us) client_negotiation.cc:769] Sending connection context
0923 20:13:59.296466 (+ 97us) client_negotiation.cc:240] Negotiation successful
0923 20:13:59.296587 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3487,"mutex_wait_us":4,"thread_start_us":3311,"threads_started":1}
I20260923 20:13:59.297622 20477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:13:59.284206 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43050 (local address 127.1.170.131:37953)
0923 20:13:59.288760 (+ 4554us) server_negotiation.cc:206] Beginning negotiation
0923 20:13:59.288770 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0923 20:13:59.289364 (+ 594us) server_negotiation.cc:407] Connection header received
0923 20:13:59.289461 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:13:59.289469 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:13:59.289697 (+ 228us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:13:59.290457 (+ 760us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:13:59.290649 (+ 192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:13:59.291739 (+ 1090us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:13:59.294610 (+ 2871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:13:59.295958 (+ 1348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:13:59.296089 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:13:59.296305 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0923 20:13:59.296543 (+ 238us) server_negotiation.cc:299] Negotiation successful
0923 20:13:59.297256 (+ 713us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4371,"thread_start_us":4205,"threads_started":1}
I20260923 20:13:59.297411 20263 tablet_service.cc:1467] Processing CreateTablet for tablet 6ea7753a4f7c410cb171dccb5413efd2 (DEFAULT_TABLE table=test-table [id=5e3b6c403069492a96f487926211f384]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 20:13:59.297282 20129 tablet_service.cc:1467] Processing CreateTablet for tablet 6ea7753a4f7c410cb171dccb5413efd2 (DEFAULT_TABLE table=test-table [id=5e3b6c403069492a96f487926211f384]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 20:13:59.298799 20129 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6ea7753a4f7c410cb171dccb5413efd2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:13:59.298803 20263 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6ea7753a4f7c410cb171dccb5413efd2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:13:59.300702 20396 tablet_service.cc:1467] Processing CreateTablet for tablet 6ea7753a4f7c410cb171dccb5413efd2 (DEFAULT_TABLE table=test-table [id=5e3b6c403069492a96f487926211f384]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 20:13:59.301945 20396 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6ea7753a4f7c410cb171dccb5413efd2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:13:59.313800 20479 tablet_bootstrap.cc:492] T 6ea7753a4f7c410cb171dccb5413efd2 P 9de5527c801840c8ba994331b1e36b4e: Bootstrap starting.
I20260923 20:13:59.316874 20480 tablet_bootstrap.cc:492] T 6ea7753a4f7c410cb171dccb5413efd2 P 49f35fde25484d6483b1d067e92ceeac: Bootstrap starting.
I20260923 20:13:59.319522 20479 tablet_bootstrap.cc:654] T 6ea7753a4f7c410cb171dccb5413efd2 P 9de5527c801840c8ba994331b1e36b4e: Neither blocks nor log segments found. Creating new log.
I20260923 20:13:59.320752 20479 log.cc:824] T 6ea7753a4f7c410cb171dccb5413efd2 P 9de5527c801840c8ba994331b1e36b4e: Log is configured to *not* fsync() on all Append() calls
I20260923 20:13:59.321182 20480 tablet_bootstrap.cc:654] T 6ea7753a4f7c410cb171dccb5413efd2 P 49f35fde25484d6483b1d067e92ceeac: Neither blocks nor log segments found. Creating new log.
I20260923 20:13:59.322515 20480 log.cc:824] T 6ea7753a4f7c410cb171dccb5413efd2 P 49f35fde25484d6483b1d067e92ceeac: Log is configured to *not* fsync() on all Append() calls
I20260923 20:13:59.323159 20481 tablet_bootstrap.cc:492] T 6ea7753a4f7c410cb171dccb5413efd2 P 5b602cdfb97d403191baafbca741cbcf: Bootstrap starting.
I20260923 20:13:59.324729 20479 tablet_bootstrap.cc:492] T 6ea7753a4f7c410cb171dccb5413efd2 P 9de5527c801840c8ba994331b1e36b4e: No bootstrap required, opened a new log
I20260923 20:13:59.325006 20479 ts_tablet_manager.cc:1397] T 6ea7753a4f7c410cb171dccb5413efd2 P 9de5527c801840c8ba994331b1e36b4e: Time spent bootstrapping tablet: real 0.012s user 0.000s sys 0.007s
I20260923 20:13:59.326254 20480 tablet_bootstrap.cc:492] T 6ea7753a4f7c410cb171dccb5413efd2 P 49f35fde25484d6483b1d067e92ceeac: No bootstrap required, opened a new log
I20260923 20:13:59.326541 20480 ts_tablet_manager.cc:1397] T 6ea7753a4f7c410cb171dccb5413efd2 P 49f35fde25484d6483b1d067e92ceeac: Time spent bootstrapping tablet: real 0.010s user 0.009s sys 0.000s
I20260923 20:13:59.326691 20481 tablet_bootstrap.cc:654] T 6ea7753a4f7c410cb171dccb5413efd2 P 5b602cdfb97d403191baafbca741cbcf: Neither blocks nor log segments found. Creating new log.
I20260923 20:13:59.327957 20481 log.cc:824] T 6ea7753a4f7c410cb171dccb5413efd2 P 5b602cdfb97d403191baafbca741cbcf: Log is configured to *not* fsync() on all Append() calls
I20260923 20:13:59.331408 20481 tablet_bootstrap.cc:492] T 6ea7753a4f7c410cb171dccb5413efd2 P 5b602cdfb97d403191baafbca741cbcf: No bootstrap required, opened a new log
I20260923 20:13:59.331692 20481 ts_tablet_manager.cc:1397] T 6ea7753a4f7c410cb171dccb5413efd2 P 5b602cdfb97d403191baafbca741cbcf: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260923 20:13:59.334199 20479 raft_consensus.cc:348] T 6ea7753a4f7c410cb171dccb5413efd2 P 9de5527c801840c8ba994331b1e36b4e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b602cdfb97d403191baafbca741cbcf" member_type: VOTER last_known_addr { host: "127.1.170.130" port: 40877 } } peers { permanent_uuid: "9de5527c801840c8ba994331b1e36b4e" member_type: VOTER last_known_addr { host: "127.1.170.129" port: 43531 } } peers { permanent_uuid: "49f35fde25484d6483b1d067e92ceeac" member_type: VOTER last_known_addr { host: "127.1.170.131" port: 37953 } }
I20260923 20:13:59.334621 20479 raft_consensus.cc:374] T 6ea7753a4f7c410cb171dccb5413efd2 P 9de5527c801840c8ba994331b1e36b4e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:13:59.334724 20479 raft_consensus.cc:729] T 6ea7753a4f7c410cb171dccb5413efd2 P 9de5527c801840c8ba994331b1e36b4e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9de5527c801840c8ba994331b1e36b4e, State: Initialized, Role: FOLLOWER
I20260923 20:13:59.335387 20479 consensus_queue.cc:260] T 6ea7753a4f7c410cb171dccb5413efd2 P 9de5527c801840c8ba994331b1e36b4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b602cdfb97d403191baafbca741cbcf" member_type: VOTER last_known_addr { host: "127.1.170.130" port: 40877 } } peers { permanent_uuid: "9de5527c801840c8ba994331b1e36b4e" member_type: VOTER last_known_addr { host: "127.1.170.129" port: 43531 } } peers { permanent_uuid: "49f35fde25484d6483b1d067e92ceeac" member_type: VOTER last_known_addr { host: "127.1.170.131" port: 37953 } }
I20260923 20:13:59.337735 20480 raft_consensus.cc:348] T 6ea7753a4f7c410cb171dccb5413efd2 P 49f35fde25484d6483b1d067e92ceeac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b602cdfb97d403191baafbca741cbcf" member_type: VOTER last_known_addr { host: "127.1.170.130" port: 40877 } } peers { permanent_uuid: "9de5527c801840c8ba994331b1e36b4e" member_type: VOTER last_known_addr { host: "127.1.170.129" port: 43531 } } peers { permanent_uuid: "49f35fde25484d6483b1d067e92ceeac" member_type: VOTER last_known_addr { host: "127.1.170.131" port: 37953 } }
I20260923 20:13:59.338325 20480 raft_consensus.cc:374] T 6ea7753a4f7c410cb171dccb5413efd2 P 49f35fde25484d6483b1d067e92ceeac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:13:59.338356 20479 ts_tablet_manager.cc:1428] T 6ea7753a4f7c410cb171dccb5413efd2 P 9de5527c801840c8ba994331b1e36b4e: Time spent starting tablet: real 0.013s user 0.013s sys 0.000s
I20260923 20:13:59.338430 20480 raft_consensus.cc:729] T 6ea7753a4f7c410cb171dccb5413efd2 P 49f35fde25484d6483b1d067e92ceeac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49f35fde25484d6483b1d067e92ceeac, State: Initialized, Role: FOLLOWER
I20260923 20:13:59.339215 20480 consensus_queue.cc:260] T 6ea7753a4f7c410cb171dccb5413efd2 P 49f35fde25484d6483b1d067e92ceeac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b602cdfb97d403191baafbca741cbcf" member_type: VOTER last_known_addr { host: "127.1.170.130" port: 40877 } } peers { permanent_uuid: "9de5527c801840c8ba994331b1e36b4e" member_type: VOTER last_known_addr { host: "127.1.170.129" port: 43531 } } peers { permanent_uuid: "49f35fde25484d6483b1d067e92ceeac" member_type: VOTER last_known_addr { host: "127.1.170.131" port: 37953 } }
I20260923 20:13:59.340435 20461 heartbeater.cc:503] Master 127.1.170.190:43945 was elected leader, sending a full tablet report...
I20260923 20:13:59.341518 20480 ts_tablet_manager.cc:1428] T 6ea7753a4f7c410cb171dccb5413efd2 P 49f35fde25484d6483b1d067e92ceeac: Time spent starting tablet: real 0.015s user 0.014s sys 0.000s
I20260923 20:13:59.343524 20481 raft_consensus.cc:348] T 6ea7753a4f7c410cb171dccb5413efd2 P 5b602cdfb97d403191baafbca741cbcf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b602cdfb97d403191baafbca741cbcf" member_type: VOTER last_known_addr { host: "127.1.170.130" port: 40877 } } peers { permanent_uuid: "9de5527c801840c8ba994331b1e36b4e" member_type: VOTER last_known_addr { host: "127.1.170.129" port: 43531 } } peers { permanent_uuid: "49f35fde25484d6483b1d067e92ceeac" member_type: VOTER last_known_addr { host: "127.1.170.131" port: 37953 } }
I20260923 20:13:59.344004 20481 raft_consensus.cc:374] T 6ea7753a4f7c410cb171dccb5413efd2 P 5b602cdfb97d403191baafbca741cbcf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:13:59.344141 20481 raft_consensus.cc:729] T 6ea7753a4f7c410cb171dccb5413efd2 P 5b602cdfb97d403191baafbca741cbcf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b602cdfb97d403191baafbca741cbcf, State: Initialized, Role: FOLLOWER
I20260923 20:13:59.344838 20481 consensus_queue.cc:260] T 6ea7753a4f7c410cb171dccb5413efd2 P 5b602cdfb97d403191baafbca741cbcf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b602cdfb97d403191baafbca741cbcf" member_type: VOTER last_known_addr { host: "127.1.170.130" port: 40877 } } peers { permanent_uuid: "9de5527c801840c8ba994331b1e36b4e" member_type: VOTER last_known_addr { host: "127.1.170.129" port: 43531 } } peers { permanent_uuid: "49f35fde25484d6483b1d067e92ceeac" member_type: VOTER last_known_addr { host: "127.1.170.131" port: 37953 } }
I20260923 20:13:59.346038 20328 heartbeater.cc:503] Master 127.1.170.190:43945 was elected leader, sending a full tablet report...
I20260923 20:13:59.346889 20481 ts_tablet_manager.cc:1428] T 6ea7753a4f7c410cb171dccb5413efd2 P 5b602cdfb97d403191baafbca741cbcf: Time spent starting tablet: real 0.015s user 0.011s sys 0.004s
W20260923 20:13:59.428028 20195 tablet.cc:2369] T 6ea7753a4f7c410cb171dccb5413efd2 P 9de5527c801840c8ba994331b1e36b4e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260923 20:13:59.439730 20329 tablet.cc:2369] T 6ea7753a4f7c410cb171dccb5413efd2 P 5b602cdfb97d403191baafbca741cbcf: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260923 20:13:59.444080 20462 tablet.cc:2369] T 6ea7753a4f7c410cb171dccb5413efd2 P 49f35fde25484d6483b1d067e92ceeac: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 20:13:59.551085 20486 raft_consensus.cc:482] T 6ea7753a4f7c410cb171dccb5413efd2 P 49f35fde25484d6483b1d067e92ceeac [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 20:13:59.551366 20486 raft_consensus.cc:504] T 6ea7753a4f7c410cb171dccb5413efd2 P 49f35fde25484d6483b1d067e92ceeac [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b602cdfb97d403191baafbca741cbcf" member_type: VOTER last_known_addr { host: "127.1.170.130" port: 40877 } } peers { permanent_uuid: "9de5527c801840c8ba994331b1e36b4e" member_type: VOTER last_known_addr { host: "127.1.170.129" port: 43531 } } peers { permanent_uuid: "49f35fde25484d6483b1d067e92ceeac" member_type: VOTER last_known_addr { host: "127.1.170.131" port: 37953 } }
I20260923 20:13:59.553051 20486 leader_election.cc:288] T 6ea7753a4f7c410cb171dccb5413efd2 P 49f35fde25484d6483b1d067e92ceeac [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5b602cdfb97d403191baafbca741cbcf (127.1.170.130:40877), 9de5527c801840c8ba994331b1e36b4e (127.1.170.129:43531)
I20260923 20:13:59.559298 20475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:13:59.553592 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.170.131:34363 (local address 127.1.170.130:40877)
0923 20:13:59.553776 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0923 20:13:59.553783 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:13:59.553865 (+ 82us) server_negotiation.cc:407] Connection header received
0923 20:13:59.554193 (+ 328us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:13:59.554200 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:13:59.554295 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:13:59.554403 (+ 108us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:13:59.554797 (+ 394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:13:59.555701 (+ 904us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:13:59.557310 (+ 1609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:13:59.558375 (+ 1065us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:13:59.558563 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:13:59.558641 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0923 20:13:59.558923 (+ 282us) server_negotiation.cc:299] Negotiation successful
0923 20:13:59.559051 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260923 20:13:59.559327 20463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:13:59.553398 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.130:40877 (local address 127.1.170.131:34363)
0923 20:13:59.553742 (+ 344us) negotiation.cc:107] Waiting for socket to connect
0923 20:13:59.553769 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0923 20:13:59.553972 (+ 203us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:13:59.554430 (+ 458us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:13:59.554435 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:13:59.554498 (+ 63us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:13:59.554665 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:13:59.554674 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:13:59.555851 (+ 1177us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:13:59.555859 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:13:59.557147 (+ 1288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:13:59.557159 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:13:59.558538 (+ 1379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:13:59.558546 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:13:59.558749 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:13:59.558774 (+ 25us) client_negotiation.cc:769] Sending connection context
0923 20:13:59.558892 (+ 118us) client_negotiation.cc:240] Negotiation successful
0923 20:13:59.559032 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":175,"spinlock_wait_cycles":58752}
I20260923 20:13:59.559813 20476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:13:59.553544 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.170.131:44361 (local address 127.1.170.129:43531)
0923 20:13:59.553758 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0923 20:13:59.553764 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:13:59.554427 (+ 663us) server_negotiation.cc:407] Connection header received
0923 20:13:59.554656 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:13:59.554660 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:13:59.554718 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:13:59.554814 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:13:59.555519 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:13:59.556173 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:13:59.557788 (+ 1615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:13:59.558649 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:13:59.558830 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:13:59.558905 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0923 20:13:59.559188 (+ 283us) server_negotiation.cc:299] Negotiation successful
0923 20:13:59.559316 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260923 20:13:59.559832 20488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:13:59.553466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.129:43531 (local address 127.1.170.131:44361)
0923 20:13:59.554320 (+ 854us) negotiation.cc:107] Waiting for socket to connect
0923 20:13:59.554335 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0923 20:13:59.554516 (+ 181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:13:59.554827 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:13:59.554831 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:13:59.554853 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:13:59.555297 (+ 444us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:13:59.555307 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:13:59.556323 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:13:59.556328 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:13:59.557634 (+ 1306us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:13:59.557645 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:13:59.558868 (+ 1223us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:13:59.558874 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:13:59.559033 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:13:59.559056 (+ 23us) client_negotiation.cc:769] Sending connection context
0923 20:13:59.559153 (+ 97us) client_negotiation.cc:240] Negotiation successful
0923 20:13:59.559281 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":732,"spinlock_wait_cycles":4224,"thread_start_us":533,"threads_started":1}
I20260923 20:13:59.560576 20283 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ea7753a4f7c410cb171dccb5413efd2" candidate_uuid: "49f35fde25484d6483b1d067e92ceeac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b602cdfb97d403191baafbca741cbcf" is_pre_election: true
I20260923 20:13:59.560801 20149 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ea7753a4f7c410cb171dccb5413efd2" candidate_uuid: "49f35fde25484d6483b1d067e92ceeac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9de5527c801840c8ba994331b1e36b4e" is_pre_election: true
I20260923 20:13:59.561116 20283 raft_consensus.cc:2454] T 6ea7753a4f7c410cb171dccb5413efd2 P 5b602cdfb97d403191baafbca741cbcf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 49f35fde25484d6483b1d067e92ceeac in term 0.
I20260923 20:13:59.561318 20149 raft_consensus.cc:2454] T 6ea7753a4f7c410cb171dccb5413efd2 P 9de5527c801840c8ba994331b1e36b4e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 49f35fde25484d6483b1d067e92ceeac in term 0.
I20260923 20:13:59.561823 20350 leader_election.cc:302] T 6ea7753a4f7c410cb171dccb5413efd2 P 49f35fde25484d6483b1d067e92ceeac [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49f35fde25484d6483b1d067e92ceeac, 5b602cdfb97d403191baafbca741cbcf; no voters:
I20260923 20:13:59.562301 20486 raft_consensus.cc:2781] T 6ea7753a4f7c410cb171dccb5413efd2 P 49f35fde25484d6483b1d067e92ceeac [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 20:13:59.562458 20486 raft_consensus.cc:482] T 6ea7753a4f7c410cb171dccb5413efd2 P 49f35fde25484d6483b1d067e92ceeac [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 20:13:59.562566 20486 raft_consensus.cc:3037] T 6ea7753a4f7c410cb171dccb5413efd2 P 49f35fde25484d6483b1d067e92ceeac [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:13:59.564704 20486 raft_consensus.cc:504] T 6ea7753a4f7c410cb171dccb5413efd2 P 49f35fde25484d6483b1d067e92ceeac [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b602cdfb97d403191baafbca741cbcf" member_type: VOTER last_known_addr { host: "127.1.170.130" port: 40877 } } peers { permanent_uuid: "9de5527c801840c8ba994331b1e36b4e" member_type: VOTER last_known_addr { host: "127.1.170.129" port: 43531 } } peers { permanent_uuid: "49f35fde25484d6483b1d067e92ceeac" member_type: VOTER last_known_addr { host: "127.1.170.131" port: 37953 } }
I20260923 20:13:59.565348 20486 leader_election.cc:288] T 6ea7753a4f7c410cb171dccb5413efd2 P 49f35fde25484d6483b1d067e92ceeac [CANDIDATE]: Term 1 election: Requested vote from peers 5b602cdfb97d403191baafbca741cbcf (127.1.170.130:40877), 9de5527c801840c8ba994331b1e36b4e (127.1.170.129:43531)
I20260923 20:13:59.565824 20283 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ea7753a4f7c410cb171dccb5413efd2" candidate_uuid: "49f35fde25484d6483b1d067e92ceeac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b602cdfb97d403191baafbca741cbcf"
I20260923 20:13:59.566052 20149 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ea7753a4f7c410cb171dccb5413efd2" candidate_uuid: "49f35fde25484d6483b1d067e92ceeac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9de5527c801840c8ba994331b1e36b4e"
I20260923 20:13:59.566095 20283 raft_consensus.cc:3037] T 6ea7753a4f7c410cb171dccb5413efd2 P 5b602cdfb97d403191baafbca741cbcf [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:13:59.566289 20149 raft_consensus.cc:3037] T 6ea7753a4f7c410cb171dccb5413efd2 P 9de5527c801840c8ba994331b1e36b4e [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:13:59.568372 20283 raft_consensus.cc:2454] T 6ea7753a4f7c410cb171dccb5413efd2 P 5b602cdfb97d403191baafbca741cbcf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 49f35fde25484d6483b1d067e92ceeac in term 1.
I20260923 20:13:59.568836 20350 leader_election.cc:302] T 6ea7753a4f7c410cb171dccb5413efd2 P 49f35fde25484d6483b1d067e92ceeac [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49f35fde25484d6483b1d067e92ceeac, 5b602cdfb97d403191baafbca741cbcf; no voters:
I20260923 20:13:59.569001 20149 raft_consensus.cc:2454] T 6ea7753a4f7c410cb171dccb5413efd2 P 9de5527c801840c8ba994331b1e36b4e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 49f35fde25484d6483b1d067e92ceeac in term 1.
I20260923 20:13:59.569131 20486 raft_consensus.cc:2781] T 6ea7753a4f7c410cb171dccb5413efd2 P 49f35fde25484d6483b1d067e92ceeac [term 1 FOLLOWER]: Leader election won for term 1
I20260923 20:13:59.569684 20486 raft_consensus.cc:686] T 6ea7753a4f7c410cb171dccb5413efd2 P 49f35fde25484d6483b1d067e92ceeac [term 1 LEADER]: Becoming Leader. State: Replica: 49f35fde25484d6483b1d067e92ceeac, State: Running, Role: LEADER
I20260923 20:13:59.570257 20486 consensus_queue.cc:237] T 6ea7753a4f7c410cb171dccb5413efd2 P 49f35fde25484d6483b1d067e92ceeac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b602cdfb97d403191baafbca741cbcf" member_type: VOTER last_known_addr { host: "127.1.170.130" port: 40877 } } peers { permanent_uuid: "9de5527c801840c8ba994331b1e36b4e" member_type: VOTER last_known_addr { host: "127.1.170.129" port: 43531 } } peers { permanent_uuid: "49f35fde25484d6483b1d067e92ceeac" member_type: VOTER last_known_addr { host: "127.1.170.131" port: 37953 } }
I20260923 20:13:59.575033 20003 catalog_manager.cc:5495] T 6ea7753a4f7c410cb171dccb5413efd2 P 49f35fde25484d6483b1d067e92ceeac reported cstate change: term changed from 0 to 1, leader changed from <none> to 49f35fde25484d6483b1d067e92ceeac (127.1.170.131). New cstate: current_term: 1 leader_uuid: "49f35fde25484d6483b1d067e92ceeac" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b602cdfb97d403191baafbca741cbcf" member_type: VOTER last_known_addr { host: "127.1.170.130" port: 40877 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9de5527c801840c8ba994331b1e36b4e" member_type: VOTER last_known_addr { host: "127.1.170.129" port: 43531 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "49f35fde25484d6483b1d067e92ceeac" member_type: VOTER last_known_addr { host: "127.1.170.131" port: 37953 } health_report { overall_health: HEALTHY } } }
I20260923 20:13:59.602427 20477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:13:59.597019 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43058 (local address 127.1.170.131:37953)
0923 20:13:59.597231 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0923 20:13:59.597240 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 20:13:59.597263 (+ 23us) server_negotiation.cc:407] Connection header received
0923 20:13:59.597359 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:13:59.597363 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:13:59.597477 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:13:59.597641 (+ 164us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 20:13:59.598110 (+ 469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:13:59.598840 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:13:59.600233 (+ 1393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:13:59.600631 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:13:59.600762 (+ 131us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:13:59.601329 (+ 567us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 20:13:59.601825 (+ 496us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 20:13:59.602056 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0923 20:13:59.602178 (+ 122us) server_negotiation.cc:299] Negotiation successful
0923 20:13:59.602217 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260923 20:13:59.610994 20283 raft_consensus.cc:1261] T 6ea7753a4f7c410cb171dccb5413efd2 P 5b602cdfb97d403191baafbca741cbcf [term 1 FOLLOWER]: Refusing update from remote peer 49f35fde25484d6483b1d067e92ceeac: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 20:13:59.610994 20149 raft_consensus.cc:1261] T 6ea7753a4f7c410cb171dccb5413efd2 P 9de5527c801840c8ba994331b1e36b4e [term 1 FOLLOWER]: Refusing update from remote peer 49f35fde25484d6483b1d067e92ceeac: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 20:13:59.612015 20489 consensus_queue.cc:1035] T 6ea7753a4f7c410cb171dccb5413efd2 P 49f35fde25484d6483b1d067e92ceeac [LEADER]: Connected to new peer: Peer: permanent_uuid: "9de5527c801840c8ba994331b1e36b4e" member_type: VOTER last_known_addr { host: "127.1.170.129" port: 43531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 20:13:59.612382 20486 consensus_queue.cc:1035] T 6ea7753a4f7c410cb171dccb5413efd2 P 49f35fde25484d6483b1d067e92ceeac [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b602cdfb97d403191baafbca741cbcf" member_type: VOTER last_known_addr { host: "127.1.170.130" port: 40877 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 20:13:59.624562 20492 mvcc.cc:205] Tried to move back new op lower bound from 7332636424635244544 to 7332636424484638720. Current Snapshot: MvccSnapshot[applied={T|T < 7332636424635244544}]
I20260923 20:13:59.642028 20003 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35332:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 20:13:59.642344 20003 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35332:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 20:13:59.646450 20003 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 447a080aa83c4a28ad854bd8b9059b4d: Sending DeleteTablet for 3 replicas of tablet 6ea7753a4f7c410cb171dccb5413efd2
I20260923 20:13:59.647743 20263 tablet_service.cc:1514] Processing DeleteTablet for tablet 6ea7753a4f7c410cb171dccb5413efd2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 20:13:59 UTC) from {username='kudu', principal='kudu/127.1.170.190@KRBTEST.COM'} at 127.0.0.1:54200
I20260923 20:13:59.647745 20129 tablet_service.cc:1514] Processing DeleteTablet for tablet 6ea7753a4f7c410cb171dccb5413efd2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 20:13:59 UTC) from {username='kudu', principal='kudu/127.1.170.190@KRBTEST.COM'} at 127.0.0.1:49718
I20260923 20:13:59.648679 20502 tablet_replica.cc:330] T 6ea7753a4f7c410cb171dccb5413efd2 P 5b602cdfb97d403191baafbca741cbcf: stopping tablet replica
I20260923 20:13:59.648845 20396 tablet_service.cc:1514] Processing DeleteTablet for tablet 6ea7753a4f7c410cb171dccb5413efd2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 20:13:59 UTC) from {username='kudu', principal='kudu/127.1.170.190@KRBTEST.COM'} at 127.0.0.1:43050
I20260923 20:13:59.649250 20502 raft_consensus.cc:2229] T 6ea7753a4f7c410cb171dccb5413efd2 P 5b602cdfb97d403191baafbca741cbcf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 20:13:59.649802 20502 raft_consensus.cc:2258] T 6ea7753a4f7c410cb171dccb5413efd2 P 5b602cdfb97d403191baafbca741cbcf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 20:13:59.650660 20503 tablet_replica.cc:330] T 6ea7753a4f7c410cb171dccb5413efd2 P 9de5527c801840c8ba994331b1e36b4e: stopping tablet replica
I20260923 20:13:59.651492 1706 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskqUuku6/build/asan/bin/kudu pid 20066
I20260923 20:13:59.652390 20502 ts_tablet_manager.cc:1905] T 6ea7753a4f7c410cb171dccb5413efd2 P 5b602cdfb97d403191baafbca741cbcf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 20:13:59.659507 20502 ts_tablet_manager.cc:1918] T 6ea7753a4f7c410cb171dccb5413efd2 P 5b602cdfb97d403191baafbca741cbcf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 20:13:59.660001 20503 raft_consensus.cc:2229] T 6ea7753a4f7c410cb171dccb5413efd2 P 9de5527c801840c8ba994331b1e36b4e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 20:13:59.660627 20503 raft_consensus.cc:2258] T 6ea7753a4f7c410cb171dccb5413efd2 P 9de5527c801840c8ba994331b1e36b4e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 20:13:59.661418 20504 tablet_replica.cc:330] T 6ea7753a4f7c410cb171dccb5413efd2 P 49f35fde25484d6483b1d067e92ceeac: stopping tablet replica
I20260923 20:13:59.661779 20504 raft_consensus.cc:2229] T 6ea7753a4f7c410cb171dccb5413efd2 P 49f35fde25484d6483b1d067e92ceeac [term 1 LEADER]: Raft consensus shutting down.
I20260923 20:13:59.662441 20504 raft_consensus.cc:2258] T 6ea7753a4f7c410cb171dccb5413efd2 P 49f35fde25484d6483b1d067e92ceeac [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 20:13:59.662940 20503 ts_tablet_manager.cc:1905] T 6ea7753a4f7c410cb171dccb5413efd2 P 9de5527c801840c8ba994331b1e36b4e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 20:13:59.666041 20502 log.cc:1196] T 6ea7753a4f7c410cb171dccb5413efd2 P 5b602cdfb97d403191baafbca741cbcf: Deleting WAL directory at /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790194121577027-1706-0/minicluster-data/ts-1/wal/wals/6ea7753a4f7c410cb171dccb5413efd2
I20260923 20:13:59.666638 20502 ts_tablet_manager.cc:1939] T 6ea7753a4f7c410cb171dccb5413efd2 P 5b602cdfb97d403191baafbca741cbcf: Deleting consensus metadata
I20260923 20:13:59.667793 20504 ts_tablet_manager.cc:1905] T 6ea7753a4f7c410cb171dccb5413efd2 P 49f35fde25484d6483b1d067e92ceeac: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 20:13:59.668908 19990 catalog_manager.cc:4841] TS 5b602cdfb97d403191baafbca741cbcf (127.1.170.130:40877): tablet 6ea7753a4f7c410cb171dccb5413efd2 (table test-table [id=5e3b6c403069492a96f487926211f384]) successfully deleted
Sep 23 20:13:59 dist-test-slave-th4g krb5kdc[19960](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194437, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.170.129@KRBTEST.COM
I20260923 20:13:59.673612 20503 ts_tablet_manager.cc:1918] T 6ea7753a4f7c410cb171dccb5413efd2 P 9de5527c801840c8ba994331b1e36b4e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 20:13:59.673998 20503 log.cc:1196] T 6ea7753a4f7c410cb171dccb5413efd2 P 9de5527c801840c8ba994331b1e36b4e: Deleting WAL directory at /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790194121577027-1706-0/minicluster-data/ts-0/wal/wals/6ea7753a4f7c410cb171dccb5413efd2
I20260923 20:13:59.674669 20503 ts_tablet_manager.cc:1939] T 6ea7753a4f7c410cb171dccb5413efd2 P 9de5527c801840c8ba994331b1e36b4e: Deleting consensus metadata
I20260923 20:13:59.675395 20504 ts_tablet_manager.cc:1918] T 6ea7753a4f7c410cb171dccb5413efd2 P 49f35fde25484d6483b1d067e92ceeac: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 20:13:59.675570 20504 log.cc:1196] T 6ea7753a4f7c410cb171dccb5413efd2 P 49f35fde25484d6483b1d067e92ceeac: Deleting WAL directory at /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790194121577027-1706-0/minicluster-data/ts-2/wal/wals/6ea7753a4f7c410cb171dccb5413efd2
I20260923 20:13:59.676182 20504 ts_tablet_manager.cc:1939] T 6ea7753a4f7c410cb171dccb5413efd2 P 49f35fde25484d6483b1d067e92ceeac: Deleting consensus metadata
I20260923 20:13:59.677098 19991 catalog_manager.cc:4841] TS 9de5527c801840c8ba994331b1e36b4e (127.1.170.129:43531): tablet 6ea7753a4f7c410cb171dccb5413efd2 (table test-table [id=5e3b6c403069492a96f487926211f384]) successfully deleted
I20260923 20:13:59.677855 19990 catalog_manager.cc:4841] TS 49f35fde25484d6483b1d067e92ceeac (127.1.170.131:37953): tablet 6ea7753a4f7c410cb171dccb5413efd2 (table test-table [id=5e3b6c403069492a96f487926211f384]) successfully deleted
I20260923 20:13:59.681407 20476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:13:59.653306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49728 (local address 127.1.170.129:43531)
0923 20:13:59.653529 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0923 20:13:59.653539 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0923 20:13:59.660913 (+ 7374us) server_negotiation.cc:407] Connection header received
0923 20:13:59.661089 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:13:59.661097 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:13:59.661211 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:13:59.661312 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:13:59.662940 (+ 1628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:13:59.664041 (+ 1101us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:13:59.665982 (+ 1941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:13:59.666592 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:13:59.666735 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:13:59.673359 (+ 6624us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:13:59.673376 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:13:59.673399 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:13:59.673464 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:13:59.678399 (+ 4935us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:13:59.679369 (+ 970us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:13:59.679376 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:13:59.679379 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:13:59.679530 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:13:59.679937 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:13:59.679941 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:13:59.679943 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:13:59.680294 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:13:59.680519 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0923 20:13:59.680895 (+ 376us) server_negotiation.cc:299] Negotiation successful
0923 20:13:59.681059 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260923 20:13:59.682406 20189 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:13:59.784400 1706 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqUuku6/build/asan/bin/kudu with pid 20066
I20260923 20:13:59.803414 1706 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskqUuku6/build/asan/bin/kudu pid 20200
Sep 23 20:13:59 dist-test-slave-th4g krb5kdc[19960](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194437, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.170.130@KRBTEST.COM
I20260923 20:13:59.822280 20475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:13:59.804280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54214 (local address 127.1.170.130:40877)
0923 20:13:59.804465 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0923 20:13:59.804478 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0923 20:13:59.804511 (+ 33us) server_negotiation.cc:407] Connection header received
0923 20:13:59.804706 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:13:59.804710 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:13:59.804782 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:13:59.804876 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:13:59.806685 (+ 1809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:13:59.807484 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:13:59.809375 (+ 1891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:13:59.809772 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:13:59.810053 (+ 281us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:13:59.815867 (+ 5814us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:13:59.815883 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:13:59.815901 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:13:59.815949 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:13:59.819111 (+ 3162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:13:59.819949 (+ 838us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:13:59.819956 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:13:59.819958 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:13:59.820092 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:13:59.820576 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:13:59.820583 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:13:59.820586 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:13:59.821016 (+ 430us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:13:59.821249 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0923 20:13:59.821643 (+ 394us) server_negotiation.cc:299] Negotiation successful
0923 20:13:59.821818 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260923 20:13:59.822973 20323 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:13:59.915549 1706 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqUuku6/build/asan/bin/kudu with pid 20200
I20260923 20:13:59.934587 1706 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskqUuku6/build/asan/bin/kudu pid 20333
Sep 23 20:13:59 dist-test-slave-th4g krb5kdc[19960](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194437, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.170.131@KRBTEST.COM
I20260923 20:13:59.955680 20477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:13:59.935701 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43068 (local address 127.1.170.131:37953)
0923 20:13:59.935932 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0923 20:13:59.935941 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 20:13:59.935964 (+ 23us) server_negotiation.cc:407] Connection header received
0923 20:13:59.936061 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:13:59.936067 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:13:59.936147 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:13:59.936280 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:13:59.937637 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:13:59.939297 (+ 1660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:13:59.940832 (+ 1535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:13:59.941609 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:13:59.941770 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:13:59.947024 (+ 5254us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:13:59.947051 (+ 27us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:13:59.947073 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:13:59.947129 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:13:59.951355 (+ 4226us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:13:59.952497 (+ 1142us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:13:59.952509 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:13:59.952518 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:13:59.952783 (+ 265us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:13:59.953461 (+ 678us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:13:59.953472 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:13:59.953476 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:13:59.954173 (+ 697us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:13:59.954561 (+ 388us) server_negotiation.cc:1035] Waiting for connection context
0923 20:13:59.954934 (+ 373us) server_negotiation.cc:299] Negotiation successful
0923 20:13:59.955193 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260923 20:13:59.956583 20456 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:14:00.083954 1706 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqUuku6/build/asan/bin/kudu with pid 20333
I20260923 20:14:00.104913 1706 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskqUuku6/build/asan/bin/kudu pid 19973
I20260923 20:14:00.106145 20035 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:14:00.218878 1706 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqUuku6/build/asan/bin/kudu with pid 19973
2026-09-23T20:14:00Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (4487 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-09-23T20:14:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T20:14:00Z Disabled control of system clock
I20260923 20:14:00.298700 1706 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqUuku6/build/asan/bin/kudu
/tmp/dist-test-taskqUuku6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790194121577027-1706-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790194121577027-1706-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790194121577027-1706-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790194121577027-1706-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.170.190:45783
--webserver_interface=127.1.170.190
--webserver_port=0
--builtin_ntp_servers=127.1.170.148:41117
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.170.190:45783
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260923 20:14:00.647006 20513 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:14:00.647351 20513 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:14:00.647514 20513 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:14:00.656644 20513 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 20:14:00.656800 20513 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:14:00.656839 20513 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:14:00.656962 20513 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 20:14:00.657059 20513 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 20:14:00.667623 20513 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.170.148:41117
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790194121577027-1706-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790194121577027-1706-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.170.190:45783
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.170.190:45783
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790194121577027-1706-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.170.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790194121577027-1706-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:14:00.669032 20513 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:14:00.670960 20513 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:14:00.686581 20519 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:14:00.686545 20518 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:14:00.686630 20521 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:14:00.688395 20513 server_base.cc:1034] running on GCE node
I20260923 20:14:00.689397 20513 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:14:00.691046 20513 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:14:00.692408 20513 hybrid_clock.cc:648] HybridClock initialized: now 1790194440692336 us; error 66 us; skew 500 ppm
I20260923 20:14:00.693076 20513 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260923 20:14:00.696389 20513 webserver.cc:460] Webserver started at http://127.1.170.190:37231/ using document root <none> and password file <none>
I20260923 20:14:00.697121 20513 fs_manager.cc:359] Metadata directory not provided
I20260923 20:14:00.697232 20513 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:14:00.697510 20513 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:14:00.700922 20513 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790194121577027-1706-0/minicluster-data/master-0/data/instance:
uuid: "cb7a0d2325054fe9b007623a71cea124"
format_stamp: "Formatted at 2026-09-23 20:14:00 on dist-test-slave-th4g"
server_key: "6498ce011b736489b99967ce814a1c78"
server_key_iv: "58f0fb252a9a9c17b4e3fdc2e4741a04"
server_key_version: "encryptionkey@0"
I20260923 20:14:00.701730 20513 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790194121577027-1706-0/minicluster-data/master-0/wal/instance:
uuid: "cb7a0d2325054fe9b007623a71cea124"
format_stamp: "Formatted at 2026-09-23 20:14:00 on dist-test-slave-th4g"
server_key: "6498ce011b736489b99967ce814a1c78"
server_key_iv: "58f0fb252a9a9c17b4e3fdc2e4741a04"
server_key_version: "encryptionkey@0"
I20260923 20:14:00.707455 20513 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.003s
I20260923 20:14:00.711444 20527 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:14:00.713186 20513 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260923 20:14:00.713390 20513 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790194121577027-1706-0/minicluster-data/master-0/data,/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790194121577027-1706-0/minicluster-data/master-0/wal
uuid: "cb7a0d2325054fe9b007623a71cea124"
format_stamp: "Formatted at 2026-09-23 20:14:00 on dist-test-slave-th4g"
server_key: "6498ce011b736489b99967ce814a1c78"
server_key_iv: "58f0fb252a9a9c17b4e3fdc2e4741a04"
server_key_version: "encryptionkey@0"
I20260923 20:14:00.713647 20513 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790194121577027-1706-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790194121577027-1706-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790194121577027-1706-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:14:00.742302 20513 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:14:00.743219 20513 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:14:00.743536 20513 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:14:00.764324 20513 rpc_server.cc:307] RPC server started. Bound to: 127.1.170.190:45783
I20260923 20:14:00.764351 20578 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.170.190:45783 every 8 connection(s)
I20260923 20:14:00.765736 20513 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790194121577027-1706-0/minicluster-data/master-0/data/info.pb
I20260923 20:14:00.770704 20579 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:14:00.774446 1706 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqUuku6/build/asan/bin/kudu as pid 20513
I20260923 20:14:00.774735 1706 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790194121577027-1706-0/minicluster-data/master-0/wal/instance
I20260923 20:14:00.775235 1706 external_mini_cluster.cc:1428] Setting key 4eb2e42b31594ea393b34de4ab603652
I20260923 20:14:00.785051 20579 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cb7a0d2325054fe9b007623a71cea124: Bootstrap starting.
I20260923 20:14:00.788976 20579 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cb7a0d2325054fe9b007623a71cea124: Neither blocks nor log segments found. Creating new log.
I20260923 20:14:00.790400 20579 log.cc:824] T 00000000000000000000000000000000 P cb7a0d2325054fe9b007623a71cea124: Log is configured to *not* fsync() on all Append() calls
I20260923 20:14:00.791385 20582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:14:00.777252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56842 (local address 127.1.170.190:45783)
0923 20:14:00.778344 (+ 1092us) server_negotiation.cc:206] Beginning negotiation
0923 20:14:00.778393 (+ 49us) server_negotiation.cc:399] Waiting for connection header
0923 20:14:00.778450 (+ 57us) server_negotiation.cc:407] Connection header received
0923 20:14:00.780353 (+ 1903us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:14:00.780401 (+ 48us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:14:00.781076 (+ 675us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:14:00.781557 (+ 481us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:14:00.782172 (+ 615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:14:00.784146 (+ 1974us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:14:00.785801 (+ 1655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:14:00.786899 (+ 1098us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:14:00.787105 (+ 206us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:14:00.788291 (+ 1186us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:14:00.788301 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:14:00.788321 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 20:14:00.789067 (+ 746us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:14:00.789189 (+ 122us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 20:14:00.789218 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:14:00.789356 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0923 20:14:00.789600 (+ 244us) server_negotiation.cc:299] Negotiation successful
0923 20:14:00.789966 (+ 366us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":581,"thread_start_us":328,"threads_started":1}
I20260923 20:14:00.794242 20579 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cb7a0d2325054fe9b007623a71cea124: No bootstrap required, opened a new log
I20260923 20:14:00.802999 20579 raft_consensus.cc:348] T 00000000000000000000000000000000 P cb7a0d2325054fe9b007623a71cea124 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb7a0d2325054fe9b007623a71cea124" member_type: VOTER last_known_addr { host: "127.1.170.190" port: 45783 } }
I20260923 20:14:00.803468 20579 raft_consensus.cc:374] T 00000000000000000000000000000000 P cb7a0d2325054fe9b007623a71cea124 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:14:00.803573 20579 raft_consensus.cc:729] T 00000000000000000000000000000000 P cb7a0d2325054fe9b007623a71cea124 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb7a0d2325054fe9b007623a71cea124, State: Initialized, Role: FOLLOWER
I20260923 20:14:00.804208 20579 consensus_queue.cc:260] T 00000000000000000000000000000000 P cb7a0d2325054fe9b007623a71cea124 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb7a0d2325054fe9b007623a71cea124" member_type: VOTER last_known_addr { host: "127.1.170.190" port: 45783 } }
I20260923 20:14:00.804471 20579 raft_consensus.cc:388] T 00000000000000000000000000000000 P cb7a0d2325054fe9b007623a71cea124 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 20:14:00.804607 20579 raft_consensus.cc:482] T 00000000000000000000000000000000 P cb7a0d2325054fe9b007623a71cea124 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 20:14:00.804785 20579 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cb7a0d2325054fe9b007623a71cea124 [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:14:00.807346 20579 raft_consensus.cc:504] T 00000000000000000000000000000000 P cb7a0d2325054fe9b007623a71cea124 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb7a0d2325054fe9b007623a71cea124" member_type: VOTER last_known_addr { host: "127.1.170.190" port: 45783 } }
I20260923 20:14:00.808063 20579 leader_election.cc:302] T 00000000000000000000000000000000 P cb7a0d2325054fe9b007623a71cea124 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cb7a0d2325054fe9b007623a71cea124; no voters:
I20260923 20:14:00.808780 20579 leader_election.cc:288] T 00000000000000000000000000000000 P cb7a0d2325054fe9b007623a71cea124 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 20:14:00.808957 20584 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cb7a0d2325054fe9b007623a71cea124 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 20:14:00.810254 20584 raft_consensus.cc:686] T 00000000000000000000000000000000 P cb7a0d2325054fe9b007623a71cea124 [term 1 LEADER]: Becoming Leader. State: Replica: cb7a0d2325054fe9b007623a71cea124, State: Running, Role: LEADER
I20260923 20:14:00.810823 20584 consensus_queue.cc:237] T 00000000000000000000000000000000 P cb7a0d2325054fe9b007623a71cea124 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb7a0d2325054fe9b007623a71cea124" member_type: VOTER last_known_addr { host: "127.1.170.190" port: 45783 } }
I20260923 20:14:00.811280 20579 sys_catalog.cc:564] T 00000000000000000000000000000000 P cb7a0d2325054fe9b007623a71cea124 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 20:14:00.813903 20586 sys_catalog.cc:455] T 00000000000000000000000000000000 P cb7a0d2325054fe9b007623a71cea124 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cb7a0d2325054fe9b007623a71cea124. Latest consensus state: current_term: 1 leader_uuid: "cb7a0d2325054fe9b007623a71cea124" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb7a0d2325054fe9b007623a71cea124" member_type: VOTER last_known_addr { host: "127.1.170.190" port: 45783 } } }
I20260923 20:14:00.813938 20585 sys_catalog.cc:455] T 00000000000000000000000000000000 P cb7a0d2325054fe9b007623a71cea124 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cb7a0d2325054fe9b007623a71cea124" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb7a0d2325054fe9b007623a71cea124" member_type: VOTER last_known_addr { host: "127.1.170.190" port: 45783 } } }
I20260923 20:14:00.814436 20586 sys_catalog.cc:458] T 00000000000000000000000000000000 P cb7a0d2325054fe9b007623a71cea124 [sys.catalog]: This master's current role is: LEADER
I20260923 20:14:00.814441 20585 sys_catalog.cc:458] T 00000000000000000000000000000000 P cb7a0d2325054fe9b007623a71cea124 [sys.catalog]: This master's current role is: LEADER
I20260923 20:14:00.815652 20593 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 20:14:00.821499 20593 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 20:14:00.830572 20593 catalog_manager.cc:1347] Generated new cluster ID: ed1c10c6cec74425941d38fbe700efd4
I20260923 20:14:00.830713 20593 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 20:14:00.850354 20593 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 20:14:00.851689 20593 catalog_manager.cc:1504] Loading token signing keys...
I20260923 20:14:00.865154 20593 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cb7a0d2325054fe9b007623a71cea124: Generated new TSK 0
I20260923 20:14:00.866132 20593 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 20:14:00.876518 1706 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqUuku6/build/asan/bin/kudu
/tmp/dist-test-taskqUuku6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790194121577027-1706-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790194121577027-1706-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790194121577027-1706-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790194121577027-1706-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.170.129:0
--local_ip_for_outbound_sockets=127.1.170.129
--webserver_interface=127.1.170.129
--webserver_port=0
--tserver_master_addrs=127.1.170.190:45783
--builtin_ntp_servers=127.1.170.148:41117
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260923 20:14:01.231684 20603 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:14:01.232023 20603 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:14:01.232115 20603 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:14:01.240453 20603 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:14:01.240571 20603 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:14:01.240682 20603 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.170.129
I20260923 20:14:01.250727 20603 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.170.148:41117
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790194121577027-1706-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790194121577027-1706-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.170.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790194121577027-1706-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.170.129
--webserver_port=0
--tserver_master_addrs=127.1.170.190:45783
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.170.129
--log_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790194121577027-1706-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:14:01.251971 20603 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:14:01.253669 20603 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:14:01.266964 20609 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:14:01.267076 20608 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:14:01.267076 20611 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:14:01.267809 20603 server_base.cc:1034] running on GCE node
I20260923 20:14:01.268391 20603 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:14:01.269322 20603 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:14:01.270550 20603 hybrid_clock.cc:648] HybridClock initialized: now 1790194441270482 us; error 60 us; skew 500 ppm
I20260923 20:14:01.270942 20603 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260923 20:14:01.273329 20603 webserver.cc:460] Webserver started at http://127.1.170.129:44883/ using document root <none> and password file <none>
I20260923 20:14:01.274102 20603 fs_manager.cc:359] Metadata directory not provided
I20260923 20:14:01.274196 20603 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:14:01.274474 20603 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:14:01.277300 20603 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790194121577027-1706-0/minicluster-data/ts-0/data/instance:
uuid: "c5d7b573013b45ca8c524d46c3dc13f5"
format_stamp: "Formatted at 2026-09-23 20:14:01 on dist-test-slave-th4g"
server_key: "eb8592c9f2de19b369eda9295488389a"
server_key_iv: "bdf173ee0c35d05a697a43e5513d56f1"
server_key_version: "encryptionkey@0"
I20260923 20:14:01.278123 20603 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790194121577027-1706-0/minicluster-data/ts-0/wal/instance:
uuid: "c5d7b573013b45ca8c524d46c3dc13f5"
format_stamp: "Formatted at 2026-09-23 20:14:01 on dist-test-slave-th4g"
server_key: "eb8592c9f2de19b369eda9295488389a"
server_key_iv: "bdf173ee0c35d05a697a43e5513d56f1"
server_key_version: "encryptionkey@0"
I20260923 20:14:01.283515 20603 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260923 20:14:01.287443 20617 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:14:01.288826 20603 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260923 20:14:01.289052 20603 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790194121577027-1706-0/minicluster-data/ts-0/data,/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790194121577027-1706-0/minicluster-data/ts-0/wal
uuid: "c5d7b573013b45ca8c524d46c3dc13f5"
format_stamp: "Formatted at 2026-09-23 20:14:01 on dist-test-slave-th4g"
server_key: "eb8592c9f2de19b369eda9295488389a"
server_key_iv: "bdf173ee0c35d05a697a43e5513d56f1"
server_key_version: "encryptionkey@0"
I20260923 20:14:01.289252 20603 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790194121577027-1706-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790194121577027-1706-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790194121577027-1706-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:14:01.310421 20603 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:14:01.311388 20603 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:14:01.311702 20603 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:14:01.313169 20603 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 20:14:01.314960 20603 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 20:14:01.315101 20603 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 20:14:01.315228 20603 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 20:14:01.315327 20603 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 20:14:01.351264 20603 rpc_server.cc:307] RPC server started. Bound to: 127.1.170.129:37531
I20260923 20:14:01.351352 20729 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.170.129:37531 every 8 connection(s)
I20260923 20:14:01.352531 20603 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790194121577027-1706-0/minicluster-data/ts-0/data/info.pb
I20260923 20:14:01.362555 1706 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqUuku6/build/asan/bin/kudu as pid 20603
I20260923 20:14:01.362740 1706 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790194121577027-1706-0/minicluster-data/ts-0/wal/instance
I20260923 20:14:01.363196 1706 external_mini_cluster.cc:1428] Setting key c1afb8e3d8f4339943c783037ea212b0
I20260923 20:14:01.364876 20732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:14:01.355086 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.170.129:48915 (local address 127.1.170.190:45783)
0923 20:14:01.355566 (+ 480us) server_negotiation.cc:206] Beginning negotiation
0923 20:14:01.355571 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 20:14:01.356505 (+ 934us) server_negotiation.cc:407] Connection header received
0923 20:14:01.358332 (+ 1827us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:14:01.358337 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:14:01.358427 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:14:01.358549 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:14:01.359628 (+ 1079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:14:01.360348 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:14:01.361937 (+ 1589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:14:01.362392 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:14:01.362526 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:14:01.363988 (+ 1462us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:14:01.363993 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:14:01.363997 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 20:14:01.364019 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:14:01.364062 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 20:14:01.364074 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:14:01.364171 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0923 20:14:01.364399 (+ 228us) server_negotiation.cc:299] Negotiation successful
0923 20:14:01.364490 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":369,"thread_start_us":269,"threads_started":1}
I20260923 20:14:01.365989 20733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:14:01.355503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.190:45783 (local address 127.1.170.129:48915)
0923 20:14:01.356361 (+ 858us) negotiation.cc:107] Waiting for socket to connect
0923 20:14:01.356400 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0923 20:14:01.358067 (+ 1667us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:14:01.358786 (+ 719us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:14:01.358794 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:14:01.358885 (+ 91us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:14:01.359466 (+ 581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:14:01.359483 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:14:01.360509 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:14:01.360512 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:14:01.361775 (+ 1263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:14:01.361787 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:14:01.362614 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:14:01.362624 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:14:01.362907 (+ 283us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:14:01.363605 (+ 698us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 20:14:01.363608 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:14:01.363646 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 20:14:01.363753 (+ 107us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:14:01.364182 (+ 429us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:14:01.364193 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:14:01.364202 (+ 9us) client_negotiation.cc:769] Sending connection context
0923 20:14:01.364363 (+ 161us) client_negotiation.cc:240] Negotiation successful
0923 20:14:01.364767 (+ 404us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":447,"thread_start_us":267,"threads_started":1}
I20260923 20:14:01.366259 1706 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqUuku6/build/asan/bin/kudu
/tmp/dist-test-taskqUuku6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790194121577027-1706-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790194121577027-1706-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790194121577027-1706-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790194121577027-1706-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.170.130:0
--local_ip_for_outbound_sockets=127.1.170.130
--webserver_interface=127.1.170.130
--webserver_port=0
--tserver_master_addrs=127.1.170.190:45783
--builtin_ntp_servers=127.1.170.148:41117
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260923 20:14:01.367650 20730 heartbeater.cc:347] Connected to a master server at 127.1.170.190:45783
I20260923 20:14:01.367975 20730 heartbeater.cc:464] Registering TS with master...
I20260923 20:14:01.368851 20730 heartbeater.cc:511] Master 127.1.170.190:45783 requested a full tablet report, sending...
I20260923 20:14:01.370976 20544 ts_manager.cc:194] Registered new tserver with Master: c5d7b573013b45ca8c524d46c3dc13f5 (127.1.170.129:37531)
I20260923 20:14:01.372422 20544 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.170.129:48915
W20260923 20:14:01.692973 20734 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:14:01.693223 20734 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:14:01.693311 20734 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:14:01.700419 20734 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:14:01.700515 20734 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:14:01.700618 20734 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.170.130
I20260923 20:14:01.709466 20734 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.170.148:41117
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790194121577027-1706-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790194121577027-1706-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.170.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790194121577027-1706-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.170.130
--webserver_port=0
--tserver_master_addrs=127.1.170.190:45783
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.170.130
--log_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790194121577027-1706-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:14:01.710671 20734 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:14:01.712505 20734 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:14:01.725862 20739 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:14:01.725816 20742 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:14:01.725837 20740 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:14:01.726796 20734 server_base.cc:1034] running on GCE node
I20260923 20:14:01.727380 20734 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:14:01.728283 20734 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:14:01.729483 20734 hybrid_clock.cc:648] HybridClock initialized: now 1790194441729435 us; error 40 us; skew 500 ppm
I20260923 20:14:01.729758 20734 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260923 20:14:01.732240 20734 webserver.cc:460] Webserver started at http://127.1.170.130:35833/ using document root <none> and password file <none>
I20260923 20:14:01.732900 20734 fs_manager.cc:359] Metadata directory not provided
I20260923 20:14:01.732992 20734 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:14:01.733253 20734 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:14:01.736069 20734 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790194121577027-1706-0/minicluster-data/ts-1/data/instance:
uuid: "5a82320ac39c4f71ad71d6966d42a4f6"
format_stamp: "Formatted at 2026-09-23 20:14:01 on dist-test-slave-th4g"
server_key: "2f71db95b785cbec6547cb1f6bb6fade"
server_key_iv: "47355212cf62a26879d50e6f02446078"
server_key_version: "encryptionkey@0"
I20260923 20:14:01.736799 20734 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790194121577027-1706-0/minicluster-data/ts-1/wal/instance:
uuid: "5a82320ac39c4f71ad71d6966d42a4f6"
format_stamp: "Formatted at 2026-09-23 20:14:01 on dist-test-slave-th4g"
server_key: "2f71db95b785cbec6547cb1f6bb6fade"
server_key_iv: "47355212cf62a26879d50e6f02446078"
server_key_version: "encryptionkey@0"
I20260923 20:14:01.741861 20734 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260923 20:14:01.745466 20748 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:14:01.747038 20734 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260923 20:14:01.747222 20734 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790194121577027-1706-0/minicluster-data/ts-1/data,/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790194121577027-1706-0/minicluster-data/ts-1/wal
uuid: "5a82320ac39c4f71ad71d6966d42a4f6"
format_stamp: "Formatted at 2026-09-23 20:14:01 on dist-test-slave-th4g"
server_key: "2f71db95b785cbec6547cb1f6bb6fade"
server_key_iv: "47355212cf62a26879d50e6f02446078"
server_key_version: "encryptionkey@0"
I20260923 20:14:01.747454 20734 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790194121577027-1706-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790194121577027-1706-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790194121577027-1706-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:14:01.775511 20734 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:14:01.776319 20734 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:14:01.776626 20734 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:14:01.777693 20734 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 20:14:01.779191 20734 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 20:14:01.779276 20734 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 20:14:01.779418 20734 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 20:14:01.779489 20734 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 20:14:01.813241 20734 rpc_server.cc:307] RPC server started. Bound to: 127.1.170.130:44345
I20260923 20:14:01.813323 20860 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.170.130:44345 every 8 connection(s)
I20260923 20:14:01.814656 20734 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790194121577027-1706-0/minicluster-data/ts-1/data/info.pb
I20260923 20:14:01.823112 1706 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqUuku6/build/asan/bin/kudu as pid 20734
I20260923 20:14:01.823268 1706 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790194121577027-1706-0/minicluster-data/ts-1/wal/instance
I20260923 20:14:01.823716 1706 external_mini_cluster.cc:1428] Setting key 055bf1bf9dafe1c64f6de135419cd0f4
I20260923 20:14:01.826853 1706 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqUuku6/build/asan/bin/kudu
/tmp/dist-test-taskqUuku6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790194121577027-1706-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790194121577027-1706-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790194121577027-1706-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790194121577027-1706-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.170.131:0
--local_ip_for_outbound_sockets=127.1.170.131
--webserver_interface=127.1.170.131
--webserver_port=0
--tserver_master_addrs=127.1.170.190:45783
--builtin_ntp_servers=127.1.170.148:41117
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260923 20:14:01.827227 20732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:14:01.817140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.170.130:56499 (local address 127.1.170.190:45783)
0923 20:14:01.817305 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0923 20:14:01.817310 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 20:14:01.818706 (+ 1396us) server_negotiation.cc:407] Connection header received
0923 20:14:01.820438 (+ 1732us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:14:01.820446 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:14:01.820534 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:14:01.820644 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:14:01.821688 (+ 1044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:14:01.822453 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:14:01.824160 (+ 1707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:14:01.824699 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:14:01.824827 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:14:01.826392 (+ 1565us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:14:01.826399 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:14:01.826403 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 20:14:01.826424 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:14:01.826463 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 20:14:01.826480 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:14:01.826574 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0923 20:14:01.826776 (+ 202us) server_negotiation.cc:299] Negotiation successful
0923 20:14:01.826874 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260923 20:14:01.828147 20863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:14:01.817622 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.190:45783 (local address 127.1.170.130:56499)
0923 20:14:01.818567 (+ 945us) negotiation.cc:107] Waiting for socket to connect
0923 20:14:01.818606 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0923 20:14:01.820185 (+ 1579us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:14:01.820855 (+ 670us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:14:01.820870 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:14:01.820986 (+ 116us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:14:01.821526 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:14:01.821542 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:14:01.822625 (+ 1083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:14:01.822629 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:14:01.824024 (+ 1395us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:14:01.824036 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:14:01.824920 (+ 884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:14:01.824928 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:14:01.825238 (+ 310us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:14:01.826063 (+ 825us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 20:14:01.826067 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:14:01.826126 (+ 59us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 20:14:01.826211 (+ 85us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:14:01.826584 (+ 373us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:14:01.826592 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:14:01.826597 (+ 5us) client_negotiation.cc:769] Sending connection context
0923 20:14:01.826734 (+ 137us) client_negotiation.cc:240] Negotiation successful
0923 20:14:01.827096 (+ 362us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":470,"thread_start_us":309,"threads_started":1}
I20260923 20:14:01.829478 20861 heartbeater.cc:347] Connected to a master server at 127.1.170.190:45783
I20260923 20:14:01.829770 20861 heartbeater.cc:464] Registering TS with master...
I20260923 20:14:01.830528 20861 heartbeater.cc:511] Master 127.1.170.190:45783 requested a full tablet report, sending...
I20260923 20:14:01.831852 20544 ts_manager.cc:194] Registered new tserver with Master: 5a82320ac39c4f71ad71d6966d42a4f6 (127.1.170.130:44345)
I20260923 20:14:01.832639 20544 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.170.130:56499
W20260923 20:14:02.150978 20864 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:14:02.151288 20864 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:14:02.151432 20864 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:14:02.159049 20864 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:14:02.159153 20864 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:14:02.159351 20864 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.170.131
I20260923 20:14:02.168615 20864 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.170.148:41117
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790194121577027-1706-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790194121577027-1706-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.170.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790194121577027-1706-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.170.131
--webserver_port=0
--tserver_master_addrs=127.1.170.190:45783
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.170.131
--log_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790194121577027-1706-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:14:02.169994 20864 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:14:02.171697 20864 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:14:02.184634 20869 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:14:02.184671 20870 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:14:02.184665 20872 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:14:02.185902 20864 server_base.cc:1034] running on GCE node
I20260923 20:14:02.186571 20864 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:14:02.187552 20864 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:14:02.188779 20864 hybrid_clock.cc:648] HybridClock initialized: now 1790194442188731 us; error 44 us; skew 500 ppm
I20260923 20:14:02.189105 20864 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260923 20:14:02.191543 20864 webserver.cc:460] Webserver started at http://127.1.170.131:45185/ using document root <none> and password file <none>
I20260923 20:14:02.192250 20864 fs_manager.cc:359] Metadata directory not provided
I20260923 20:14:02.192385 20864 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:14:02.192723 20864 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:14:02.195765 20864 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790194121577027-1706-0/minicluster-data/ts-2/data/instance:
uuid: "28bdc130f2844a4680ca9fd458d58458"
format_stamp: "Formatted at 2026-09-23 20:14:02 on dist-test-slave-th4g"
server_key: "1ca2492f71a363d43374b647634bea5c"
server_key_iv: "a0b1421029b0a938b4b7c89f4339d92d"
server_key_version: "encryptionkey@0"
I20260923 20:14:02.196569 20864 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790194121577027-1706-0/minicluster-data/ts-2/wal/instance:
uuid: "28bdc130f2844a4680ca9fd458d58458"
format_stamp: "Formatted at 2026-09-23 20:14:02 on dist-test-slave-th4g"
server_key: "1ca2492f71a363d43374b647634bea5c"
server_key_iv: "a0b1421029b0a938b4b7c89f4339d92d"
server_key_version: "encryptionkey@0"
I20260923 20:14:02.201797 20864 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260923 20:14:02.205293 20878 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:14:02.206686 20864 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260923 20:14:02.206856 20864 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790194121577027-1706-0/minicluster-data/ts-2/data,/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790194121577027-1706-0/minicluster-data/ts-2/wal
uuid: "28bdc130f2844a4680ca9fd458d58458"
format_stamp: "Formatted at 2026-09-23 20:14:02 on dist-test-slave-th4g"
server_key: "1ca2492f71a363d43374b647634bea5c"
server_key_iv: "a0b1421029b0a938b4b7c89f4339d92d"
server_key_version: "encryptionkey@0"
I20260923 20:14:02.207084 20864 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790194121577027-1706-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790194121577027-1706-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790194121577027-1706-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:14:02.242794 20864 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:14:02.243666 20864 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:14:02.244001 20864 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:14:02.245222 20864 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 20:14:02.246779 20864 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 20:14:02.246901 20864 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 20:14:02.247031 20864 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 20:14:02.247116 20864 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 20:14:02.280562 20864 rpc_server.cc:307] RPC server started. Bound to: 127.1.170.131:43583
I20260923 20:14:02.280616 20990 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.170.131:43583 every 8 connection(s)
I20260923 20:14:02.281860 20864 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790194121577027-1706-0/minicluster-data/ts-2/data/info.pb
I20260923 20:14:02.291453 1706 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqUuku6/build/asan/bin/kudu as pid 20864
I20260923 20:14:02.291626 1706 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790194121577027-1706-0/minicluster-data/ts-2/wal/instance
I20260923 20:14:02.292138 1706 external_mini_cluster.cc:1428] Setting key 368863055b8949fe195e9c6d4961c076
I20260923 20:14:02.294660 20732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:14:02.284431 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.170.131:34907 (local address 127.1.170.190:45783)
0923 20:14:02.284630 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0923 20:14:02.284634 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 20:14:02.286073 (+ 1439us) server_negotiation.cc:407] Connection header received
0923 20:14:02.287790 (+ 1717us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:14:02.287795 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:14:02.287863 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:14:02.287955 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:14:02.288918 (+ 963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:14:02.289666 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:14:02.291320 (+ 1654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:14:02.291869 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:14:02.291976 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:14:02.293665 (+ 1689us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:14:02.293673 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:14:02.293679 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 20:14:02.293705 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:14:02.293738 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 20:14:02.293752 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:14:02.293862 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0923 20:14:02.294145 (+ 283us) server_negotiation.cc:299] Negotiation successful
0923 20:14:02.294245 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260923 20:14:02.295719 20993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:14:02.284947 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.190:45783 (local address 127.1.170.131:34907)
0923 20:14:02.285858 (+ 911us) negotiation.cc:107] Waiting for socket to connect
0923 20:14:02.285954 (+ 96us) client_negotiation.cc:174] Beginning negotiation
0923 20:14:02.287532 (+ 1578us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:14:02.288136 (+ 604us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:14:02.288143 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:14:02.288222 (+ 79us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:14:02.288770 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:14:02.288786 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:14:02.289805 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:14:02.289809 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:14:02.291141 (+ 1332us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:14:02.291158 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:14:02.292094 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:14:02.292101 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:14:02.292428 (+ 327us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:14:02.293212 (+ 784us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 20:14:02.293217 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:14:02.293291 (+ 74us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 20:14:02.293398 (+ 107us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:14:02.293861 (+ 463us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:14:02.293910 (+ 49us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:14:02.293918 (+ 8us) client_negotiation.cc:769] Sending connection context
0923 20:14:02.294127 (+ 209us) client_negotiation.cc:240] Negotiation successful
0923 20:14:02.294533 (+ 406us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":497,"thread_start_us":301,"threads_started":1}
I20260923 20:14:02.297093 20991 heartbeater.cc:347] Connected to a master server at 127.1.170.190:45783
I20260923 20:14:02.297439 20991 heartbeater.cc:464] Registering TS with master...
I20260923 20:14:02.298273 20991 heartbeater.cc:511] Master 127.1.170.190:45783 requested a full tablet report, sending...
I20260923 20:14:02.299837 20544 ts_manager.cc:194] Registered new tserver with Master: 28bdc130f2844a4680ca9fd458d58458 (127.1.170.131:43583)
I20260923 20:14:02.300727 20544 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.170.131:34907
I20260923 20:14:02.307754 1706 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260923 20:14:02.318204 20732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:14:02.311982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56858 (local address 127.1.170.190:45783)
0923 20:14:02.312139 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0923 20:14:02.312146 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:14:02.312581 (+ 435us) server_negotiation.cc:407] Connection header received
0923 20:14:02.312841 (+ 260us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:14:02.312845 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:14:02.312922 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:14:02.313008 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:14:02.313415 (+ 407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:14:02.314261 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:14:02.315795 (+ 1534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:14:02.316211 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:14:02.316353 (+ 142us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:14:02.317389 (+ 1036us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:14:02.317394 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:14:02.317397 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 20:14:02.317414 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:14:02.317447 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 20:14:02.317458 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:14:02.317567 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0923 20:14:02.317861 (+ 294us) server_negotiation.cc:299] Negotiation successful
0923 20:14:02.317970 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260923 20:14:02.322302 20544 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56858:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 20:14:02.329422 20544 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 20:14:02.367769 21005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:14:02.342073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57958 (local address 127.1.170.131:43583)
0923 20:14:02.358393 (+ 16320us) server_negotiation.cc:206] Beginning negotiation
0923 20:14:02.358404 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0923 20:14:02.358443 (+ 39us) server_negotiation.cc:407] Connection header received
0923 20:14:02.358536 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:14:02.358543 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:14:02.358789 (+ 246us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:14:02.359700 (+ 911us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:14:02.360053 (+ 353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:14:02.362033 (+ 1980us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:14:02.364827 (+ 2794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:14:02.366416 (+ 1589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:14:02.366566 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:14:02.366847 (+ 281us) server_negotiation.cc:1035] Waiting for connection context
0923 20:14:02.367224 (+ 377us) server_negotiation.cc:299] Negotiation successful
0923 20:14:02.367414 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16150,"thread_start_us":15995,"threads_started":1}
I20260923 20:14:02.367774 21003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:14:02.341822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.131:43583 (local address 127.0.0.1:57958)
0923 20:14:02.356876 (+ 15054us) negotiation.cc:107] Waiting for socket to connect
0923 20:14:02.356922 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0923 20:14:02.357092 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:14:02.359020 (+ 1928us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:14:02.359027 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:14:02.359059 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:14:02.359467 (+ 408us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:14:02.359483 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:14:02.362163 (+ 2680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:14:02.362169 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:14:02.364588 (+ 2419us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:14:02.364604 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:14:02.366630 (+ 2026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:14:02.366638 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:14:02.366980 (+ 342us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:14:02.367039 (+ 59us) client_negotiation.cc:769] Sending connection context
0923 20:14:02.367175 (+ 136us) client_negotiation.cc:240] Negotiation successful
0923 20:14:02.367377 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14777,"mutex_wait_us":4,"thread_start_us":15417,"threads_started":1}
I20260923 20:14:02.369510 21004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:14:02.342384 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.129:37531 (local address 127.0.0.1:39816)
0923 20:14:02.361699 (+ 19315us) negotiation.cc:107] Waiting for socket to connect
0923 20:14:02.361715 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0923 20:14:02.361853 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:14:02.362517 (+ 664us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:14:02.362522 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:14:02.362555 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:14:02.363194 (+ 639us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:14:02.363207 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:14:02.364857 (+ 1650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:14:02.364862 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:14:02.366832 (+ 1970us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:14:02.366850 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:14:02.368691 (+ 1841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:14:02.368697 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:14:02.368852 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:14:02.368883 (+ 31us) client_negotiation.cc:769] Sending connection context
0923 20:14:02.368980 (+ 97us) client_negotiation.cc:240] Negotiation successful
0923 20:14:02.369099 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":19202,"mutex_wait_us":262,"thread_start_us":18730,"threads_started":1}
I20260923 20:14:02.369544 21008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:14:02.343423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39816 (local address 127.1.170.129:37531)
0923 20:14:02.360519 (+ 17096us) server_negotiation.cc:206] Beginning negotiation
0923 20:14:02.360532 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0923 20:14:02.361825 (+ 1293us) server_negotiation.cc:407] Connection header received
0923 20:14:02.362078 (+ 253us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:14:02.362083 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:14:02.362313 (+ 230us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:14:02.362507 (+ 194us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:14:02.363374 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:14:02.364693 (+ 1319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:14:02.367044 (+ 2351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:14:02.368501 (+ 1457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:14:02.368642 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:14:02.368892 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0923 20:14:02.369020 (+ 128us) server_negotiation.cc:299] Negotiation successful
0923 20:14:02.369174 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16904,"thread_start_us":16771,"threads_started":1}
I20260923 20:14:02.370432 21006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:14:02.341913 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.130:44345 (local address 127.0.0.1:41552)
0923 20:14:02.361406 (+ 19493us) negotiation.cc:107] Waiting for socket to connect
0923 20:14:02.361431 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0923 20:14:02.361588 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:14:02.362394 (+ 806us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:14:02.362397 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:14:02.362416 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:14:02.363032 (+ 616us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:14:02.363048 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:14:02.364896 (+ 1848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:14:02.364902 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:14:02.367271 (+ 2369us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:14:02.367288 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:14:02.369395 (+ 2107us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:14:02.369401 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:14:02.369601 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:14:02.369636 (+ 35us) client_negotiation.cc:769] Sending connection context
0923 20:14:02.369742 (+ 106us) client_negotiation.cc:240] Negotiation successful
0923 20:14:02.369896 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":19347,"mutex_wait_us":34,"thread_start_us":19542,"threads_started":1}
I20260923 20:14:02.371253 21007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:14:02.343012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41552 (local address 127.1.170.130:44345)
0923 20:14:02.360938 (+ 17926us) server_negotiation.cc:206] Beginning negotiation
0923 20:14:02.360949 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0923 20:14:02.361673 (+ 724us) server_negotiation.cc:407] Connection header received
0923 20:14:02.361769 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:14:02.361776 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:14:02.362016 (+ 240us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:14:02.362194 (+ 178us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:14:02.363250 (+ 1056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:14:02.364740 (+ 1490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:14:02.367492 (+ 2752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:14:02.368927 (+ 1435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:14:02.369054 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:14:02.369257 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0923 20:14:02.370721 (+ 1464us) server_negotiation.cc:299] Negotiation successful
0923 20:14:02.370907 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17761,"thread_start_us":17668,"threads_started":1}
I20260923 20:14:02.372042 20926 tablet_service.cc:1467] Processing CreateTablet for tablet db5a37b15b064db0acd0d965b1d6bf82 (DEFAULT_TABLE table=test-table [id=ae87117b58db4e3c80bcac933cecd106]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 20:14:02.373317 20926 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet db5a37b15b064db0acd0d965b1d6bf82. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:14:02.373095 20665 tablet_service.cc:1467] Processing CreateTablet for tablet db5a37b15b064db0acd0d965b1d6bf82 (DEFAULT_TABLE table=test-table [id=ae87117b58db4e3c80bcac933cecd106]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 20:14:02.374300 20665 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet db5a37b15b064db0acd0d965b1d6bf82. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:14:02.375984 20730 heartbeater.cc:503] Master 127.1.170.190:45783 was elected leader, sending a full tablet report...
I20260923 20:14:02.377317 20796 tablet_service.cc:1467] Processing CreateTablet for tablet db5a37b15b064db0acd0d965b1d6bf82 (DEFAULT_TABLE table=test-table [id=ae87117b58db4e3c80bcac933cecd106]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 20:14:02.379944 20796 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet db5a37b15b064db0acd0d965b1d6bf82. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:14:02.387099 21009 tablet_bootstrap.cc:492] T db5a37b15b064db0acd0d965b1d6bf82 P c5d7b573013b45ca8c524d46c3dc13f5: Bootstrap starting.
I20260923 20:14:02.389138 21010 tablet_bootstrap.cc:492] T db5a37b15b064db0acd0d965b1d6bf82 P 28bdc130f2844a4680ca9fd458d58458: Bootstrap starting.
I20260923 20:14:02.392011 21009 tablet_bootstrap.cc:654] T db5a37b15b064db0acd0d965b1d6bf82 P c5d7b573013b45ca8c524d46c3dc13f5: Neither blocks nor log segments found. Creating new log.
I20260923 20:14:02.392675 21010 tablet_bootstrap.cc:654] T db5a37b15b064db0acd0d965b1d6bf82 P 28bdc130f2844a4680ca9fd458d58458: Neither blocks nor log segments found. Creating new log.
I20260923 20:14:02.393384 21009 log.cc:824] T db5a37b15b064db0acd0d965b1d6bf82 P c5d7b573013b45ca8c524d46c3dc13f5: Log is configured to *not* fsync() on all Append() calls
I20260923 20:14:02.394093 21010 log.cc:824] T db5a37b15b064db0acd0d965b1d6bf82 P 28bdc130f2844a4680ca9fd458d58458: Log is configured to *not* fsync() on all Append() calls
I20260923 20:14:02.396472 21011 tablet_bootstrap.cc:492] T db5a37b15b064db0acd0d965b1d6bf82 P 5a82320ac39c4f71ad71d6966d42a4f6: Bootstrap starting.
I20260923 20:14:02.397663 21010 tablet_bootstrap.cc:492] T db5a37b15b064db0acd0d965b1d6bf82 P 28bdc130f2844a4680ca9fd458d58458: No bootstrap required, opened a new log
I20260923 20:14:02.397976 21010 ts_tablet_manager.cc:1397] T db5a37b15b064db0acd0d965b1d6bf82 P 28bdc130f2844a4680ca9fd458d58458: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.006s
I20260923 20:14:02.398175 21009 tablet_bootstrap.cc:492] T db5a37b15b064db0acd0d965b1d6bf82 P c5d7b573013b45ca8c524d46c3dc13f5: No bootstrap required, opened a new log
I20260923 20:14:02.398422 21009 ts_tablet_manager.cc:1397] T db5a37b15b064db0acd0d965b1d6bf82 P c5d7b573013b45ca8c524d46c3dc13f5: Time spent bootstrapping tablet: real 0.012s user 0.005s sys 0.005s
I20260923 20:14:02.400264 21011 tablet_bootstrap.cc:654] T db5a37b15b064db0acd0d965b1d6bf82 P 5a82320ac39c4f71ad71d6966d42a4f6: Neither blocks nor log segments found. Creating new log.
I20260923 20:14:02.401512 21011 log.cc:824] T db5a37b15b064db0acd0d965b1d6bf82 P 5a82320ac39c4f71ad71d6966d42a4f6: Log is configured to *not* fsync() on all Append() calls
I20260923 20:14:02.404793 21011 tablet_bootstrap.cc:492] T db5a37b15b064db0acd0d965b1d6bf82 P 5a82320ac39c4f71ad71d6966d42a4f6: No bootstrap required, opened a new log
I20260923 20:14:02.405086 21011 ts_tablet_manager.cc:1397] T db5a37b15b064db0acd0d965b1d6bf82 P 5a82320ac39c4f71ad71d6966d42a4f6: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260923 20:14:02.406414 21009 raft_consensus.cc:348] T db5a37b15b064db0acd0d965b1d6bf82 P c5d7b573013b45ca8c524d46c3dc13f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28bdc130f2844a4680ca9fd458d58458" member_type: VOTER last_known_addr { host: "127.1.170.131" port: 43583 } } peers { permanent_uuid: "5a82320ac39c4f71ad71d6966d42a4f6" member_type: VOTER last_known_addr { host: "127.1.170.130" port: 44345 } } peers { permanent_uuid: "c5d7b573013b45ca8c524d46c3dc13f5" member_type: VOTER last_known_addr { host: "127.1.170.129" port: 37531 } }
I20260923 20:14:02.406780 21009 raft_consensus.cc:374] T db5a37b15b064db0acd0d965b1d6bf82 P c5d7b573013b45ca8c524d46c3dc13f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:14:02.406893 21009 raft_consensus.cc:729] T db5a37b15b064db0acd0d965b1d6bf82 P c5d7b573013b45ca8c524d46c3dc13f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5d7b573013b45ca8c524d46c3dc13f5, State: Initialized, Role: FOLLOWER
I20260923 20:14:02.407493 21009 consensus_queue.cc:260] T db5a37b15b064db0acd0d965b1d6bf82 P c5d7b573013b45ca8c524d46c3dc13f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28bdc130f2844a4680ca9fd458d58458" member_type: VOTER last_known_addr { host: "127.1.170.131" port: 43583 } } peers { permanent_uuid: "5a82320ac39c4f71ad71d6966d42a4f6" member_type: VOTER last_known_addr { host: "127.1.170.130" port: 44345 } } peers { permanent_uuid: "c5d7b573013b45ca8c524d46c3dc13f5" member_type: VOTER last_known_addr { host: "127.1.170.129" port: 37531 } }
I20260923 20:14:02.408156 21010 raft_consensus.cc:348] T db5a37b15b064db0acd0d965b1d6bf82 P 28bdc130f2844a4680ca9fd458d58458 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28bdc130f2844a4680ca9fd458d58458" member_type: VOTER last_known_addr { host: "127.1.170.131" port: 43583 } } peers { permanent_uuid: "5a82320ac39c4f71ad71d6966d42a4f6" member_type: VOTER last_known_addr { host: "127.1.170.130" port: 44345 } } peers { permanent_uuid: "c5d7b573013b45ca8c524d46c3dc13f5" member_type: VOTER last_known_addr { host: "127.1.170.129" port: 37531 } }
I20260923 20:14:02.408677 21010 raft_consensus.cc:374] T db5a37b15b064db0acd0d965b1d6bf82 P 28bdc130f2844a4680ca9fd458d58458 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:14:02.408807 21010 raft_consensus.cc:729] T db5a37b15b064db0acd0d965b1d6bf82 P 28bdc130f2844a4680ca9fd458d58458 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28bdc130f2844a4680ca9fd458d58458, State: Initialized, Role: FOLLOWER
I20260923 20:14:02.409282 21009 ts_tablet_manager.cc:1428] T db5a37b15b064db0acd0d965b1d6bf82 P c5d7b573013b45ca8c524d46c3dc13f5: Time spent starting tablet: real 0.011s user 0.011s sys 0.000s
I20260923 20:14:02.409572 21010 consensus_queue.cc:260] T db5a37b15b064db0acd0d965b1d6bf82 P 28bdc130f2844a4680ca9fd458d58458 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28bdc130f2844a4680ca9fd458d58458" member_type: VOTER last_known_addr { host: "127.1.170.131" port: 43583 } } peers { permanent_uuid: "5a82320ac39c4f71ad71d6966d42a4f6" member_type: VOTER last_known_addr { host: "127.1.170.130" port: 44345 } } peers { permanent_uuid: "c5d7b573013b45ca8c524d46c3dc13f5" member_type: VOTER last_known_addr { host: "127.1.170.129" port: 37531 } }
I20260923 20:14:02.410852 20991 heartbeater.cc:503] Master 127.1.170.190:45783 was elected leader, sending a full tablet report...
I20260923 20:14:02.411406 21010 ts_tablet_manager.cc:1428] T db5a37b15b064db0acd0d965b1d6bf82 P 28bdc130f2844a4680ca9fd458d58458: Time spent starting tablet: real 0.013s user 0.013s sys 0.000s
I20260923 20:14:02.416895 21011 raft_consensus.cc:348] T db5a37b15b064db0acd0d965b1d6bf82 P 5a82320ac39c4f71ad71d6966d42a4f6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28bdc130f2844a4680ca9fd458d58458" member_type: VOTER last_known_addr { host: "127.1.170.131" port: 43583 } } peers { permanent_uuid: "5a82320ac39c4f71ad71d6966d42a4f6" member_type: VOTER last_known_addr { host: "127.1.170.130" port: 44345 } } peers { permanent_uuid: "c5d7b573013b45ca8c524d46c3dc13f5" member_type: VOTER last_known_addr { host: "127.1.170.129" port: 37531 } }
I20260923 20:14:02.417269 21011 raft_consensus.cc:374] T db5a37b15b064db0acd0d965b1d6bf82 P 5a82320ac39c4f71ad71d6966d42a4f6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:14:02.417375 21011 raft_consensus.cc:729] T db5a37b15b064db0acd0d965b1d6bf82 P 5a82320ac39c4f71ad71d6966d42a4f6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a82320ac39c4f71ad71d6966d42a4f6, State: Initialized, Role: FOLLOWER
I20260923 20:14:02.418009 21011 consensus_queue.cc:260] T db5a37b15b064db0acd0d965b1d6bf82 P 5a82320ac39c4f71ad71d6966d42a4f6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28bdc130f2844a4680ca9fd458d58458" member_type: VOTER last_known_addr { host: "127.1.170.131" port: 43583 } } peers { permanent_uuid: "5a82320ac39c4f71ad71d6966d42a4f6" member_type: VOTER last_known_addr { host: "127.1.170.130" port: 44345 } } peers { permanent_uuid: "c5d7b573013b45ca8c524d46c3dc13f5" member_type: VOTER last_known_addr { host: "127.1.170.129" port: 37531 } }
I20260923 20:14:02.419037 20861 heartbeater.cc:503] Master 127.1.170.190:45783 was elected leader, sending a full tablet report...
I20260923 20:14:02.419456 21011 ts_tablet_manager.cc:1428] T db5a37b15b064db0acd0d965b1d6bf82 P 5a82320ac39c4f71ad71d6966d42a4f6: Time spent starting tablet: real 0.014s user 0.014s sys 0.000s
I20260923 20:14:02.447942 21015 raft_consensus.cc:482] T db5a37b15b064db0acd0d965b1d6bf82 P c5d7b573013b45ca8c524d46c3dc13f5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 20:14:02.448315 21015 raft_consensus.cc:504] T db5a37b15b064db0acd0d965b1d6bf82 P c5d7b573013b45ca8c524d46c3dc13f5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28bdc130f2844a4680ca9fd458d58458" member_type: VOTER last_known_addr { host: "127.1.170.131" port: 43583 } } peers { permanent_uuid: "5a82320ac39c4f71ad71d6966d42a4f6" member_type: VOTER last_known_addr { host: "127.1.170.130" port: 44345 } } peers { permanent_uuid: "c5d7b573013b45ca8c524d46c3dc13f5" member_type: VOTER last_known_addr { host: "127.1.170.129" port: 37531 } }
I20260923 20:14:02.450120 21015 leader_election.cc:288] T db5a37b15b064db0acd0d965b1d6bf82 P c5d7b573013b45ca8c524d46c3dc13f5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 28bdc130f2844a4680ca9fd458d58458 (127.1.170.131:43583), 5a82320ac39c4f71ad71d6966d42a4f6 (127.1.170.130:44345)
I20260923 20:14:02.457445 21018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:14:02.450405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.130:44345 (local address 127.1.170.129:33787)
0923 20:14:02.451681 (+ 1276us) negotiation.cc:107] Waiting for socket to connect
0923 20:14:02.451702 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0923 20:14:02.451881 (+ 179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:14:02.452356 (+ 475us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:14:02.452361 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:14:02.452402 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:14:02.452820 (+ 418us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:14:02.452841 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:14:02.453782 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:14:02.453789 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:14:02.455409 (+ 1620us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:14:02.455421 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:14:02.456664 (+ 1243us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:14:02.456671 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:14:02.456850 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:14:02.456886 (+ 36us) client_negotiation.cc:769] Sending connection context
0923 20:14:02.456977 (+ 91us) client_negotiation.cc:240] Negotiation successful
0923 20:14:02.457077 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":875,"mutex_wait_us":39,"thread_start_us":693,"threads_started":1}
I20260923 20:14:02.457458 21007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:14:02.450675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.170.129:33787 (local address 127.1.170.130:44345)
0923 20:14:02.450956 (+ 281us) server_negotiation.cc:206] Beginning negotiation
0923 20:14:02.450965 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 20:14:02.451837 (+ 872us) server_negotiation.cc:407] Connection header received
0923 20:14:02.452099 (+ 262us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:14:02.452104 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:14:02.452210 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:14:02.452307 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:14:02.452978 (+ 671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:14:02.453638 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:14:02.455625 (+ 1987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:14:02.456471 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:14:02.456593 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:14:02.456644 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0923 20:14:02.457043 (+ 399us) server_negotiation.cc:299] Negotiation successful
0923 20:14:02.457190 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260923 20:14:02.457635 21005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:14:02.450847 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.170.129:41609 (local address 127.1.170.131:43583)
0923 20:14:02.451189 (+ 342us) server_negotiation.cc:206] Beginning negotiation
0923 20:14:02.451198 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 20:14:02.451817 (+ 619us) server_negotiation.cc:407] Connection header received
0923 20:14:02.452024 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:14:02.452029 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:14:02.452112 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:14:02.452204 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:14:02.452863 (+ 659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:14:02.453657 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:14:02.455824 (+ 2167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:14:02.456660 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:14:02.456819 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:14:02.456883 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0923 20:14:02.457236 (+ 353us) server_negotiation.cc:299] Negotiation successful
0923 20:14:02.457372 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":165}
I20260923 20:14:02.458042 21019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:14:02.450474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.131:43583 (local address 127.1.170.129:41609)
0923 20:14:02.451734 (+ 1260us) negotiation.cc:107] Waiting for socket to connect
0923 20:14:02.451749 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0923 20:14:02.451891 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:14:02.452257 (+ 366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:14:02.452265 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:14:02.452294 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:14:02.452685 (+ 391us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:14:02.452695 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:14:02.453852 (+ 1157us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:14:02.453856 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:14:02.455622 (+ 1766us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:14:02.455647 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:14:02.456877 (+ 1230us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:14:02.456881 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:14:02.457058 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:14:02.457089 (+ 31us) client_negotiation.cc:769] Sending connection context
0923 20:14:02.457189 (+ 100us) client_negotiation.cc:240] Negotiation successful
0923 20:14:02.457311 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1146,"mutex_wait_us":149,"spinlock_wait_cycles":896,"thread_start_us":939,"threads_started":1}
I20260923 20:14:02.458645 20816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db5a37b15b064db0acd0d965b1d6bf82" candidate_uuid: "c5d7b573013b45ca8c524d46c3dc13f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a82320ac39c4f71ad71d6966d42a4f6" is_pre_election: true
I20260923 20:14:02.458905 20946 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db5a37b15b064db0acd0d965b1d6bf82" candidate_uuid: "c5d7b573013b45ca8c524d46c3dc13f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28bdc130f2844a4680ca9fd458d58458" is_pre_election: true
I20260923 20:14:02.459147 20816 raft_consensus.cc:2454] T db5a37b15b064db0acd0d965b1d6bf82 P 5a82320ac39c4f71ad71d6966d42a4f6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5d7b573013b45ca8c524d46c3dc13f5 in term 0.
I20260923 20:14:02.459405 20946 raft_consensus.cc:2454] T db5a37b15b064db0acd0d965b1d6bf82 P 28bdc130f2844a4680ca9fd458d58458 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5d7b573013b45ca8c524d46c3dc13f5 in term 0.
I20260923 20:14:02.459837 20619 leader_election.cc:302] T db5a37b15b064db0acd0d965b1d6bf82 P c5d7b573013b45ca8c524d46c3dc13f5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5a82320ac39c4f71ad71d6966d42a4f6, c5d7b573013b45ca8c524d46c3dc13f5; no voters:
I20260923 20:14:02.460260 21015 raft_consensus.cc:2781] T db5a37b15b064db0acd0d965b1d6bf82 P c5d7b573013b45ca8c524d46c3dc13f5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 20:14:02.460413 21015 raft_consensus.cc:482] T db5a37b15b064db0acd0d965b1d6bf82 P c5d7b573013b45ca8c524d46c3dc13f5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 20:14:02.460532 21015 raft_consensus.cc:3037] T db5a37b15b064db0acd0d965b1d6bf82 P c5d7b573013b45ca8c524d46c3dc13f5 [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:14:02.462764 21015 raft_consensus.cc:504] T db5a37b15b064db0acd0d965b1d6bf82 P c5d7b573013b45ca8c524d46c3dc13f5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28bdc130f2844a4680ca9fd458d58458" member_type: VOTER last_known_addr { host: "127.1.170.131" port: 43583 } } peers { permanent_uuid: "5a82320ac39c4f71ad71d6966d42a4f6" member_type: VOTER last_known_addr { host: "127.1.170.130" port: 44345 } } peers { permanent_uuid: "c5d7b573013b45ca8c524d46c3dc13f5" member_type: VOTER last_known_addr { host: "127.1.170.129" port: 37531 } }
I20260923 20:14:02.463352 21015 leader_election.cc:288] T db5a37b15b064db0acd0d965b1d6bf82 P c5d7b573013b45ca8c524d46c3dc13f5 [CANDIDATE]: Term 1 election: Requested vote from peers 28bdc130f2844a4680ca9fd458d58458 (127.1.170.131:43583), 5a82320ac39c4f71ad71d6966d42a4f6 (127.1.170.130:44345)
I20260923 20:14:02.463845 20946 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db5a37b15b064db0acd0d965b1d6bf82" candidate_uuid: "c5d7b573013b45ca8c524d46c3dc13f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28bdc130f2844a4680ca9fd458d58458"
I20260923 20:14:02.463893 20816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db5a37b15b064db0acd0d965b1d6bf82" candidate_uuid: "c5d7b573013b45ca8c524d46c3dc13f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a82320ac39c4f71ad71d6966d42a4f6"
I20260923 20:14:02.464035 20946 raft_consensus.cc:3037] T db5a37b15b064db0acd0d965b1d6bf82 P 28bdc130f2844a4680ca9fd458d58458 [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:14:02.464159 20816 raft_consensus.cc:3037] T db5a37b15b064db0acd0d965b1d6bf82 P 5a82320ac39c4f71ad71d6966d42a4f6 [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:14:02.466384 20946 raft_consensus.cc:2454] T db5a37b15b064db0acd0d965b1d6bf82 P 28bdc130f2844a4680ca9fd458d58458 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c5d7b573013b45ca8c524d46c3dc13f5 in term 1.
I20260923 20:14:02.466420 20816 raft_consensus.cc:2454] T db5a37b15b064db0acd0d965b1d6bf82 P 5a82320ac39c4f71ad71d6966d42a4f6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c5d7b573013b45ca8c524d46c3dc13f5 in term 1.
I20260923 20:14:02.466816 20620 leader_election.cc:302] T db5a37b15b064db0acd0d965b1d6bf82 P c5d7b573013b45ca8c524d46c3dc13f5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28bdc130f2844a4680ca9fd458d58458, c5d7b573013b45ca8c524d46c3dc13f5; no voters:
I20260923 20:14:02.467170 21015 raft_consensus.cc:2781] T db5a37b15b064db0acd0d965b1d6bf82 P c5d7b573013b45ca8c524d46c3dc13f5 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 20:14:02.467633 21015 raft_consensus.cc:686] T db5a37b15b064db0acd0d965b1d6bf82 P c5d7b573013b45ca8c524d46c3dc13f5 [term 1 LEADER]: Becoming Leader. State: Replica: c5d7b573013b45ca8c524d46c3dc13f5, State: Running, Role: LEADER
I20260923 20:14:02.468127 21015 consensus_queue.cc:237] T db5a37b15b064db0acd0d965b1d6bf82 P c5d7b573013b45ca8c524d46c3dc13f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28bdc130f2844a4680ca9fd458d58458" member_type: VOTER last_known_addr { host: "127.1.170.131" port: 43583 } } peers { permanent_uuid: "5a82320ac39c4f71ad71d6966d42a4f6" member_type: VOTER last_known_addr { host: "127.1.170.130" port: 44345 } } peers { permanent_uuid: "c5d7b573013b45ca8c524d46c3dc13f5" member_type: VOTER last_known_addr { host: "127.1.170.129" port: 37531 } }
I20260923 20:14:02.472991 20544 catalog_manager.cc:5495] T db5a37b15b064db0acd0d965b1d6bf82 P c5d7b573013b45ca8c524d46c3dc13f5 reported cstate change: term changed from 0 to 1, leader changed from <none> to c5d7b573013b45ca8c524d46c3dc13f5 (127.1.170.129). New cstate: current_term: 1 leader_uuid: "c5d7b573013b45ca8c524d46c3dc13f5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28bdc130f2844a4680ca9fd458d58458" member_type: VOTER last_known_addr { host: "127.1.170.131" port: 43583 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5a82320ac39c4f71ad71d6966d42a4f6" member_type: VOTER last_known_addr { host: "127.1.170.130" port: 44345 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c5d7b573013b45ca8c524d46c3dc13f5" member_type: VOTER last_known_addr { host: "127.1.170.129" port: 37531 } health_report { overall_health: HEALTHY } } }
I20260923 20:14:02.518829 21008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:14:02.513363 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39828 (local address 127.1.170.129:37531)
0923 20:14:02.513543 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0923 20:14:02.513547 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 20:14:02.513619 (+ 72us) server_negotiation.cc:407] Connection header received
0923 20:14:02.513785 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:14:02.513790 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:14:02.513945 (+ 155us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:14:02.514110 (+ 165us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 20:14:02.514426 (+ 316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:14:02.515291 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:14:02.516676 (+ 1385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:14:02.517088 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:14:02.517233 (+ 145us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:14:02.517703 (+ 470us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 20:14:02.518277 (+ 574us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 20:14:02.518465 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0923 20:14:02.518556 (+ 91us) server_negotiation.cc:299] Negotiation successful
0923 20:14:02.518595 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260923 20:14:02.527503 20816 raft_consensus.cc:1261] T db5a37b15b064db0acd0d965b1d6bf82 P 5a82320ac39c4f71ad71d6966d42a4f6 [term 1 FOLLOWER]: Refusing update from remote peer c5d7b573013b45ca8c524d46c3dc13f5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 20:14:02.527501 20946 raft_consensus.cc:1261] T db5a37b15b064db0acd0d965b1d6bf82 P 28bdc130f2844a4680ca9fd458d58458 [term 1 FOLLOWER]: Refusing update from remote peer c5d7b573013b45ca8c524d46c3dc13f5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 20:14:02.528641 21020 consensus_queue.cc:1035] T db5a37b15b064db0acd0d965b1d6bf82 P c5d7b573013b45ca8c524d46c3dc13f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "28bdc130f2844a4680ca9fd458d58458" member_type: VOTER last_known_addr { host: "127.1.170.131" port: 43583 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 20:14:02.528996 21015 consensus_queue.cc:1035] T db5a37b15b064db0acd0d965b1d6bf82 P c5d7b573013b45ca8c524d46c3dc13f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a82320ac39c4f71ad71d6966d42a4f6" member_type: VOTER last_known_addr { host: "127.1.170.130" port: 44345 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260923 20:14:02.533524 20992 tablet.cc:2369] T db5a37b15b064db0acd0d965b1d6bf82 P 28bdc130f2844a4680ca9fd458d58458: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 20:14:02.540845 21023 mvcc.cc:205] Tried to move back new op lower bound from 7332636436581404672 to 7332636436353019904. Current Snapshot: MvccSnapshot[applied={T|T < 7332636436581404672}]
I20260923 20:14:02.554196 20544 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56858:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 20:14:02.554453 20544 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56858:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 20:14:02.558027 20544 catalog_manager.cc:5782] T 00000000000000000000000000000000 P cb7a0d2325054fe9b007623a71cea124: Sending DeleteTablet for 3 replicas of tablet db5a37b15b064db0acd0d965b1d6bf82
I20260923 20:14:02.559289 20796 tablet_service.cc:1514] Processing DeleteTablet for tablet db5a37b15b064db0acd0d965b1d6bf82 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 20:14:02 UTC) from {username='slave'} at 127.0.0.1:41552
I20260923 20:14:02.559444 20926 tablet_service.cc:1514] Processing DeleteTablet for tablet db5a37b15b064db0acd0d965b1d6bf82 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 20:14:02 UTC) from {username='slave'} at 127.0.0.1:57958
I20260923 20:14:02.560223 21033 tablet_replica.cc:330] T db5a37b15b064db0acd0d965b1d6bf82 P 5a82320ac39c4f71ad71d6966d42a4f6: stopping tablet replica
I20260923 20:14:02.560225 20665 tablet_service.cc:1514] Processing DeleteTablet for tablet db5a37b15b064db0acd0d965b1d6bf82 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 20:14:02 UTC) from {username='slave'} at 127.0.0.1:39816
I20260923 20:14:02.560752 21033 raft_consensus.cc:2229] T db5a37b15b064db0acd0d965b1d6bf82 P 5a82320ac39c4f71ad71d6966d42a4f6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 20:14:02.561103 1706 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskqUuku6/build/asan/bin/kudu pid 20603
I20260923 20:14:02.561334 21033 raft_consensus.cc:2258] T db5a37b15b064db0acd0d965b1d6bf82 P 5a82320ac39c4f71ad71d6966d42a4f6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 20:14:02.563701 21033 ts_tablet_manager.cc:1905] T db5a37b15b064db0acd0d965b1d6bf82 P 5a82320ac39c4f71ad71d6966d42a4f6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 20:14:02.572503 21033 ts_tablet_manager.cc:1918] T db5a37b15b064db0acd0d965b1d6bf82 P 5a82320ac39c4f71ad71d6966d42a4f6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 20:14:02.572762 21033 log.cc:1196] T db5a37b15b064db0acd0d965b1d6bf82 P 5a82320ac39c4f71ad71d6966d42a4f6: Deleting WAL directory at /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790194121577027-1706-0/minicluster-data/ts-1/wal/wals/db5a37b15b064db0acd0d965b1d6bf82
I20260923 20:14:02.573426 21033 ts_tablet_manager.cc:1939] T db5a37b15b064db0acd0d965b1d6bf82 P 5a82320ac39c4f71ad71d6966d42a4f6: Deleting consensus metadata
I20260923 20:14:02.575659 20529 catalog_manager.cc:4841] TS 5a82320ac39c4f71ad71d6966d42a4f6 (127.1.170.130:44345): tablet db5a37b15b064db0acd0d965b1d6bf82 (table test-table [id=ae87117b58db4e3c80bcac933cecd106]) successfully deleted
I20260923 20:14:02.578809 21035 tablet_replica.cc:330] T db5a37b15b064db0acd0d965b1d6bf82 P c5d7b573013b45ca8c524d46c3dc13f5: stopping tablet replica
I20260923 20:14:02.579002 21034 tablet_replica.cc:330] T db5a37b15b064db0acd0d965b1d6bf82 P 28bdc130f2844a4680ca9fd458d58458: stopping tablet replica
I20260923 20:14:02.579376 21035 raft_consensus.cc:2229] T db5a37b15b064db0acd0d965b1d6bf82 P c5d7b573013b45ca8c524d46c3dc13f5 [term 1 LEADER]: Raft consensus shutting down.
I20260923 20:14:02.579507 21034 raft_consensus.cc:2229] T db5a37b15b064db0acd0d965b1d6bf82 P 28bdc130f2844a4680ca9fd458d58458 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 20:14:02.580108 21034 raft_consensus.cc:2258] T db5a37b15b064db0acd0d965b1d6bf82 P 28bdc130f2844a4680ca9fd458d58458 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 20:14:02.580148 21035 raft_consensus.cc:2258] T db5a37b15b064db0acd0d965b1d6bf82 P c5d7b573013b45ca8c524d46c3dc13f5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 20:14:02.582278 21035 ts_tablet_manager.cc:1905] T db5a37b15b064db0acd0d965b1d6bf82 P c5d7b573013b45ca8c524d46c3dc13f5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 20:14:02.582791 21008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:14:02.562497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39840 (local address 127.1.170.129:37531)
0923 20:14:02.562694 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0923 20:14:02.562701 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:14:02.573485 (+ 10784us) server_negotiation.cc:407] Connection header received
0923 20:14:02.573993 (+ 508us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:14:02.573999 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:14:02.574102 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:14:02.574185 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:14:02.574860 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:14:02.576084 (+ 1224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:14:02.577994 (+ 1910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:14:02.579078 (+ 1084us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:14:02.579202 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:14:02.580730 (+ 1528us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:14:02.580738 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:14:02.580751 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 20:14:02.581581 (+ 830us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:14:02.581646 (+ 65us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 20:14:02.581681 (+ 35us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:14:02.581791 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0923 20:14:02.582339 (+ 548us) server_negotiation.cc:299] Negotiation successful
0923 20:14:02.582438 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260923 20:14:02.582867 21034 ts_tablet_manager.cc:1905] T db5a37b15b064db0acd0d965b1d6bf82 P 28bdc130f2844a4680ca9fd458d58458: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 20:14:02.583659 20725 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:14:02.589037 21034 ts_tablet_manager.cc:1918] T db5a37b15b064db0acd0d965b1d6bf82 P 28bdc130f2844a4680ca9fd458d58458: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 20:14:02.589203 21034 log.cc:1196] T db5a37b15b064db0acd0d965b1d6bf82 P 28bdc130f2844a4680ca9fd458d58458: Deleting WAL directory at /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790194121577027-1706-0/minicluster-data/ts-2/wal/wals/db5a37b15b064db0acd0d965b1d6bf82
I20260923 20:14:02.589622 21034 ts_tablet_manager.cc:1939] T db5a37b15b064db0acd0d965b1d6bf82 P 28bdc130f2844a4680ca9fd458d58458: Deleting consensus metadata
I20260923 20:14:02.591403 20530 catalog_manager.cc:4841] TS 28bdc130f2844a4680ca9fd458d58458 (127.1.170.131:43583): tablet db5a37b15b064db0acd0d965b1d6bf82 (table test-table [id=ae87117b58db4e3c80bcac933cecd106]) successfully deleted
I20260923 20:14:02.680428 1706 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqUuku6/build/asan/bin/kudu with pid 20603
W20260923 20:14:02.698524 20528 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.1.170.129:37531 (error 108)
I20260923 20:14:02.698750 1706 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskqUuku6/build/asan/bin/kudu pid 20734
I20260923 20:14:02.700420 21006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:14:02.700017 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.129:37531 (local address 127.0.0.1:39850)
0923 20:14:02.700178 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0923 20:14:02.700287 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.170.129:37531: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260923 20:14:02.700708 20528 catalog_manager.cc:4584] TS c5d7b573013b45ca8c524d46c3dc13f5 (127.1.170.129:37531): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet db5a37b15b064db0acd0d965b1d6bf82: Network error: Client connection negotiation failed: client connection to 127.1.170.129:37531: connect: Connection refused (error 111)
I20260923 20:14:02.705735 21007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:14:02.699528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41562 (local address 127.1.170.130:44345)
0923 20:14:02.699695 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0923 20:14:02.699701 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:14:02.699772 (+ 71us) server_negotiation.cc:407] Connection header received
0923 20:14:02.699986 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:14:02.699991 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:14:02.700075 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:14:02.700177 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:14:02.700569 (+ 392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:14:02.701348 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:14:02.702672 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:14:02.703235 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:14:02.703372 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:14:02.704423 (+ 1051us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:14:02.704455 (+ 32us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:14:02.704468 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 20:14:02.705018 (+ 550us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:14:02.705060 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 20:14:02.705079 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:14:02.705192 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0923 20:14:02.705308 (+ 116us) server_negotiation.cc:299] Negotiation successful
0923 20:14:02.705417 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260923 20:14:02.706496 20856 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:14:02.754411 21006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:14:02.753825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.129:37531 (local address 127.0.0.1:39862)
0923 20:14:02.754106 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0923 20:14:02.754214 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.170.129:37531: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130}
I20260923 20:14:02.755467 21006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:14:02.755104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.129:37531 (local address 127.0.0.1:39864)
0923 20:14:02.755249 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0923 20:14:02.755331 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.170.129:37531: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260923 20:14:02.795698 1706 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqUuku6/build/asan/bin/kudu with pid 20734
I20260923 20:14:02.813220 1706 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskqUuku6/build/asan/bin/kudu pid 20864
I20260923 20:14:02.815912 21006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:14:02.815377 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.129:37531 (local address 127.0.0.1:39866)
0923 20:14:02.815561 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0923 20:14:02.815705 (+ 144us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.170.129:37531: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260923 20:14:02.816963 21006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:14:02.816566 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.129:37531 (local address 127.0.0.1:39878)
0923 20:14:02.816732 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0923 20:14:02.816800 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.170.129:37531: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260923 20:14:02.820801 21005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:14:02.814038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57962 (local address 127.1.170.131:43583)
0923 20:14:02.814242 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0923 20:14:02.814246 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 20:14:02.814330 (+ 84us) server_negotiation.cc:407] Connection header received
0923 20:14:02.814480 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:14:02.814483 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:14:02.814568 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:14:02.814654 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:14:02.815089 (+ 435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:14:02.816334 (+ 1245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:14:02.817831 (+ 1497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:14:02.818405 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:14:02.818508 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:14:02.819509 (+ 1001us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:14:02.819518 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:14:02.819532 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 20:14:02.820110 (+ 578us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:14:02.820156 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 20:14:02.820176 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:14:02.820260 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0923 20:14:02.820385 (+ 125us) server_negotiation.cc:299] Negotiation successful
0923 20:14:02.820465 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260923 20:14:02.821429 20986 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:14:02.897431 21006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:14:02.896980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.129:37531 (local address 127.0.0.1:39892)
0923 20:14:02.897192 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0923 20:14:02.897277 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.170.129:37531: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
I20260923 20:14:02.898491 21006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:14:02.898113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.129:37531 (local address 127.0.0.1:39894)
0923 20:14:02.898312 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0923 20:14:02.898377 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.170.129:37531: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91}
I20260923 20:14:02.910080 1706 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqUuku6/build/asan/bin/kudu with pid 20864
I20260923 20:14:02.927121 1706 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskqUuku6/build/asan/bin/kudu pid 20513
I20260923 20:14:02.928222 20574 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:14:03.037994 1706 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqUuku6/build/asan/bin/kudu with pid 20513
2026-09-23T20:14:03Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (2818 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790194121577027-1706-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 20:14:03 dist-test-slave-th4g krb5kdc[21044](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 20:14:03 dist-test-slave-th4g krb5kdc[21044](info): set up 2 sockets
Sep 23 20:14:03 dist-test-slave-th4g krb5kdc[21044](info): commencing operation
krb5kdc: starting...
W20260923 20:14:04.178622 1706 mini_kdc.cc:121] Time spent starting KDC: real 1.098s user 0.000s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790194121577027-1706-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790194121577027-1706-0/krb5kdc/test-user.keytab.
Sep 23 20:14:04 dist-test-slave-th4g krb5kdc[21044](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194444, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T20:14:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T20:14:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.170.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.170.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.170.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790194121577027-1706-0/krb5kdc/kudu_127.1.170.190.keytab.
Entry for principal kudu/127.1.170.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790194121577027-1706-0/krb5kdc/kudu_127.1.170.190.keytab.
I20260923 20:14:04.458992 1706 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqUuku6/build/asan/bin/kudu
/tmp/dist-test-taskqUuku6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790194121577027-1706-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790194121577027-1706-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790194121577027-1706-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790194121577027-1706-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.170.190:43721
--webserver_interface=127.1.170.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790194121577027-1706-0/krb5kdc/kudu_127.1.170.190.keytab
--principal=kudu/127.1.170.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.170.148:34319
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.170.190:43721
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790194121577027-1706-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790194121577027-1706-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790194121577027-1706-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:14:04.808300 21057 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:14:04.808719 21057 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:14:04.808892 21057 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:14:04.816567 21057 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 20:14:04.816758 21057 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:14:04.816886 21057 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:14:04.816990 21057 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 20:14:04.817087 21057 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 20:14:04.827749 21057 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.170.148:34319
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790194121577027-1706-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790194121577027-1706-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.170.190:43721
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790194121577027-1706-0/krb5kdc/kudu_127.1.170.190.keytab
--principal=kudu/127.1.170.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.170.190:43721
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790194121577027-1706-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.170.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790194121577027-1706-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:14:04.829262 21057 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:14:04.830634 21057 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:14:04.844321 21063 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:14:04.844450 21062 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:14:04.845089 21065 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:14:04.845533 21057 server_base.cc:1034] running on GCE node
I20260923 20:14:04.847030 21057 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:14:04.848712 21057 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:14:04.850145 21057 hybrid_clock.cc:648] HybridClock initialized: now 1790194444850078 us; error 64 us; skew 500 ppm
I20260923 20:14:04.850589 21057 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:14:04 dist-test-slave-th4g krb5kdc[21044](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194444, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.170.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:14:04.854277 21057 init.cc:377] Logged in from keytab as kudu/127.1.170.190@KRBTEST.COM (short username kudu)
I20260923 20:14:04.855930 21057 webserver.cc:460] Webserver started at http://127.1.170.190:35015/ using document root <none> and password file <none>
I20260923 20:14:04.856649 21057 fs_manager.cc:359] Metadata directory not provided
I20260923 20:14:04.856770 21057 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:14:04.857052 21057 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:14:04.860431 21057 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790194121577027-1706-0/minicluster-data/master-0/data/instance:
uuid: "4f63fafbc48f4491848bc1a9e15565e1"
format_stamp: "Formatted at 2026-09-23 20:14:04 on dist-test-slave-th4g"
server_key: "0b41e3ba04496a76682c4f50824cfa86"
server_key_iv: "574cd3bcbe38ca030c25a7d5096710a9"
server_key_version: "encryptionkey@0"
I20260923 20:14:04.861205 21057 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790194121577027-1706-0/minicluster-data/master-0/wal/instance:
uuid: "4f63fafbc48f4491848bc1a9e15565e1"
format_stamp: "Formatted at 2026-09-23 20:14:04 on dist-test-slave-th4g"
server_key: "0b41e3ba04496a76682c4f50824cfa86"
server_key_iv: "574cd3bcbe38ca030c25a7d5096710a9"
server_key_version: "encryptionkey@0"
I20260923 20:14:04.866775 21057 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260923 20:14:04.871106 21072 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:14:04.872747 21057 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260923 20:14:04.872975 21057 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790194121577027-1706-0/minicluster-data/master-0/data,/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790194121577027-1706-0/minicluster-data/master-0/wal
uuid: "4f63fafbc48f4491848bc1a9e15565e1"
format_stamp: "Formatted at 2026-09-23 20:14:04 on dist-test-slave-th4g"
server_key: "0b41e3ba04496a76682c4f50824cfa86"
server_key_iv: "574cd3bcbe38ca030c25a7d5096710a9"
server_key_version: "encryptionkey@0"
I20260923 20:14:04.873199 21057 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790194121577027-1706-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790194121577027-1706-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790194121577027-1706-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:14:04.887831 21057 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:14:04.892644 21057 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:14:04.892992 21057 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:14:04.914451 21057 rpc_server.cc:307] RPC server started. Bound to: 127.1.170.190:43721
I20260923 20:14:04.914475 21123 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.170.190:43721 every 8 connection(s)
I20260923 20:14:04.915948 21057 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790194121577027-1706-0/minicluster-data/master-0/data/info.pb
I20260923 20:14:04.919996 21124 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:14:04.921458 1706 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqUuku6/build/asan/bin/kudu as pid 21057
I20260923 20:14:04.921659 1706 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790194121577027-1706-0/minicluster-data/master-0/wal/instance
I20260923 20:14:04.922284 1706 external_mini_cluster.cc:1428] Setting key 216bc9902e63405c4206657aa866d0ac
I20260923 20:14:04.937477 21124 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4f63fafbc48f4491848bc1a9e15565e1: Bootstrap starting.
Sep 23 20:14:04 dist-test-slave-th4g krb5kdc[21044](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194444, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.170.190@KRBTEST.COM
I20260923 20:14:04.941390 21124 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4f63fafbc48f4491848bc1a9e15565e1: Neither blocks nor log segments found. Creating new log.
I20260923 20:14:04.942703 21124 log.cc:824] T 00000000000000000000000000000000 P 4f63fafbc48f4491848bc1a9e15565e1: Log is configured to *not* fsync() on all Append() calls
I20260923 20:14:04.946033 21124 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4f63fafbc48f4491848bc1a9e15565e1: No bootstrap required, opened a new log
I20260923 20:14:04.951426 21127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:14:04.924374 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59694 (local address 127.1.170.190:43721)
0923 20:14:04.925400 (+ 1026us) server_negotiation.cc:206] Beginning negotiation
0923 20:14:04.925416 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0923 20:14:04.925460 (+ 44us) server_negotiation.cc:407] Connection header received
0923 20:14:04.927241 (+ 1781us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:14:04.927265 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:14:04.927739 (+ 474us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:14:04.928110 (+ 371us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:14:04.931019 (+ 2909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:14:04.932875 (+ 1856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:14:04.935103 (+ 2228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:14:04.935971 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:14:04.936139 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:14:04.941258 (+ 5119us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:14:04.941268 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:14:04.941288 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:14:04.941335 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:14:04.945494 (+ 4159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:14:04.946511 (+ 1017us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:14:04.946517 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:14:04.946525 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:14:04.946707 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:14:04.947181 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:14:04.947185 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:14:04.947187 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:14:04.948469 (+ 1282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:14:04.949015 (+ 546us) server_negotiation.cc:1035] Waiting for connection context
0923 20:14:04.949749 (+ 734us) server_negotiation.cc:299] Negotiation successful
0923 20:14:04.950276 (+ 527us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":791,"thread_start_us":537,"threads_started":1}
I20260923 20:14:04.955395 21124 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4f63fafbc48f4491848bc1a9e15565e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f63fafbc48f4491848bc1a9e15565e1" member_type: VOTER last_known_addr { host: "127.1.170.190" port: 43721 } }
I20260923 20:14:04.955811 21124 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4f63fafbc48f4491848bc1a9e15565e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:14:04.955926 21124 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4f63fafbc48f4491848bc1a9e15565e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4f63fafbc48f4491848bc1a9e15565e1, State: Initialized, Role: FOLLOWER
I20260923 20:14:04.956533 21124 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4f63fafbc48f4491848bc1a9e15565e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f63fafbc48f4491848bc1a9e15565e1" member_type: VOTER last_known_addr { host: "127.1.170.190" port: 43721 } }
I20260923 20:14:04.956790 21124 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4f63fafbc48f4491848bc1a9e15565e1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 20:14:04.956938 21124 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4f63fafbc48f4491848bc1a9e15565e1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 20:14:04.957113 21124 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4f63fafbc48f4491848bc1a9e15565e1 [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:14:04.959821 21124 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4f63fafbc48f4491848bc1a9e15565e1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f63fafbc48f4491848bc1a9e15565e1" member_type: VOTER last_known_addr { host: "127.1.170.190" port: 43721 } }
I20260923 20:14:04.960361 21124 leader_election.cc:302] T 00000000000000000000000000000000 P 4f63fafbc48f4491848bc1a9e15565e1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4f63fafbc48f4491848bc1a9e15565e1; no voters:
I20260923 20:14:04.961022 21124 leader_election.cc:288] T 00000000000000000000000000000000 P 4f63fafbc48f4491848bc1a9e15565e1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 20:14:04.961200 21129 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4f63fafbc48f4491848bc1a9e15565e1 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 20:14:04.962035 21129 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4f63fafbc48f4491848bc1a9e15565e1 [term 1 LEADER]: Becoming Leader. State: Replica: 4f63fafbc48f4491848bc1a9e15565e1, State: Running, Role: LEADER
I20260923 20:14:04.962594 21129 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4f63fafbc48f4491848bc1a9e15565e1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f63fafbc48f4491848bc1a9e15565e1" member_type: VOTER last_known_addr { host: "127.1.170.190" port: 43721 } }
I20260923 20:14:04.963254 21124 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4f63fafbc48f4491848bc1a9e15565e1 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 20:14:04.966506 21131 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4f63fafbc48f4491848bc1a9e15565e1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4f63fafbc48f4491848bc1a9e15565e1. Latest consensus state: current_term: 1 leader_uuid: "4f63fafbc48f4491848bc1a9e15565e1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f63fafbc48f4491848bc1a9e15565e1" member_type: VOTER last_known_addr { host: "127.1.170.190" port: 43721 } } }
I20260923 20:14:04.966637 21130 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4f63fafbc48f4491848bc1a9e15565e1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4f63fafbc48f4491848bc1a9e15565e1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f63fafbc48f4491848bc1a9e15565e1" member_type: VOTER last_known_addr { host: "127.1.170.190" port: 43721 } } }
I20260923 20:14:04.966943 21131 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4f63fafbc48f4491848bc1a9e15565e1 [sys.catalog]: This master's current role is: LEADER
I20260923 20:14:04.967018 21130 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4f63fafbc48f4491848bc1a9e15565e1 [sys.catalog]: This master's current role is: LEADER
I20260923 20:14:04.967905 21137 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 20:14:04.973729 21137 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 20:14:04.983680 21137 catalog_manager.cc:1347] Generated new cluster ID: e458159cd05f4c309c037929b624cf7b
I20260923 20:14:04.983811 21137 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 20:14:05.006592 21137 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 20:14:05.008656 21137 catalog_manager.cc:1504] Loading token signing keys...
I20260923 20:14:05.023049 21137 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4f63fafbc48f4491848bc1a9e15565e1: Generated new TSK 0
I20260923 20:14:05.024233 21137 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.170.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.170.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.170.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790194121577027-1706-0/krb5kdc/kudu_127.1.170.129.keytab.
Entry for principal kudu/127.1.170.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790194121577027-1706-0/krb5kdc/kudu_127.1.170.129.keytab.
I20260923 20:14:05.103055 1706 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqUuku6/build/asan/bin/kudu
/tmp/dist-test-taskqUuku6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790194121577027-1706-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790194121577027-1706-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790194121577027-1706-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790194121577027-1706-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.170.129:0
--local_ip_for_outbound_sockets=127.1.170.129
--webserver_interface=127.1.170.129
--webserver_port=0
--tserver_master_addrs=127.1.170.190:43721
--keytab_file=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790194121577027-1706-0/krb5kdc/kudu_127.1.170.129.keytab
--principal=kudu/127.1.170.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.170.148:34319
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790194121577027-1706-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790194121577027-1706-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790194121577027-1706-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:14:05.450657 21150 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:14:05.451002 21150 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:14:05.451177 21150 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:14:05.458936 21150 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:14:05.459089 21150 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:14:05.459267 21150 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.170.129
I20260923 20:14:05.469515 21150 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.170.148:34319
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790194121577027-1706-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790194121577027-1706-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790194121577027-1706-0/krb5kdc/kudu_127.1.170.129.keytab
--principal=kudu/127.1.170.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.170.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790194121577027-1706-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.170.129
--webserver_port=0
--tserver_master_addrs=127.1.170.190:43721
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.170.129
--log_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790194121577027-1706-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:14:05.470865 21150 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:14:05.472133 21150 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:14:05.486351 21156 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:14:05.486493 21158 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:14:05.486393 21155 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:14:05.487721 21150 server_base.cc:1034] running on GCE node
I20260923 20:14:05.488421 21150 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:14:05.489374 21150 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:14:05.490597 21150 hybrid_clock.cc:648] HybridClock initialized: now 1790194445490544 us; error 44 us; skew 500 ppm
I20260923 20:14:05.490928 21150 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:14:05 dist-test-slave-th4g krb5kdc[21044](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194445, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.170.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:14:05.494223 21150 init.cc:377] Logged in from keytab as kudu/127.1.170.129@KRBTEST.COM (short username kudu)
I20260923 20:14:05.495754 21150 webserver.cc:460] Webserver started at http://127.1.170.129:35195/ using document root <none> and password file <none>
I20260923 20:14:05.496457 21150 fs_manager.cc:359] Metadata directory not provided
I20260923 20:14:05.496605 21150 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:14:05.496915 21150 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:14:05.499862 21150 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790194121577027-1706-0/minicluster-data/ts-0/data/instance:
uuid: "4f69f10b722f451cb008413449c8a1a3"
format_stamp: "Formatted at 2026-09-23 20:14:05 on dist-test-slave-th4g"
server_key: "a10380dc513b81bc07c4c33648ce9f99"
server_key_iv: "3a7e55937e2d2f7da19f7d5a510b65b9"
server_key_version: "encryptionkey@0"
I20260923 20:14:05.500682 21150 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790194121577027-1706-0/minicluster-data/ts-0/wal/instance:
uuid: "4f69f10b722f451cb008413449c8a1a3"
format_stamp: "Formatted at 2026-09-23 20:14:05 on dist-test-slave-th4g"
server_key: "a10380dc513b81bc07c4c33648ce9f99"
server_key_iv: "3a7e55937e2d2f7da19f7d5a510b65b9"
server_key_version: "encryptionkey@0"
I20260923 20:14:05.505712 21150 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260923 20:14:05.509025 21165 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:14:05.510385 21150 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 20:14:05.510565 21150 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790194121577027-1706-0/minicluster-data/ts-0/data,/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790194121577027-1706-0/minicluster-data/ts-0/wal
uuid: "4f69f10b722f451cb008413449c8a1a3"
format_stamp: "Formatted at 2026-09-23 20:14:05 on dist-test-slave-th4g"
server_key: "a10380dc513b81bc07c4c33648ce9f99"
server_key_iv: "3a7e55937e2d2f7da19f7d5a510b65b9"
server_key_version: "encryptionkey@0"
I20260923 20:14:05.510769 21150 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790194121577027-1706-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790194121577027-1706-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790194121577027-1706-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:14:05.532079 21150 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:14:05.536190 21150 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:14:05.536485 21150 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:14:05.537459 21150 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 20:14:05.538892 21150 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 20:14:05.539001 21150 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 20:14:05.539147 21150 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 20:14:05.539219 21150 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 20:14:05.574070 21150 rpc_server.cc:307] RPC server started. Bound to: 127.1.170.129:44809
I20260923 20:14:05.574116 21277 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.170.129:44809 every 8 connection(s)
I20260923 20:14:05.575368 21150 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790194121577027-1706-0/minicluster-data/ts-0/data/info.pb
I20260923 20:14:05.579635 1706 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqUuku6/build/asan/bin/kudu as pid 21150
I20260923 20:14:05.579811 1706 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790194121577027-1706-0/minicluster-data/ts-0/wal/instance
I20260923 20:14:05.580380 1706 external_mini_cluster.cc:1428] Setting key 8b29aaf67b11ab962deee91c62e4b5b3
Sep 23 20:14:05 dist-test-slave-th4g krb5kdc[21044](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194445, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.170.129@KRBTEST.COM for kudu/127.1.170.190@KRBTEST.COM
I20260923 20:14:05.597787 21280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:14:05.577951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.170.129:40131 (local address 127.1.170.190:43721)
0923 20:14:05.578391 (+ 440us) server_negotiation.cc:206] Beginning negotiation
0923 20:14:05.578397 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:14:05.579466 (+ 1069us) server_negotiation.cc:407] Connection header received
0923 20:14:05.581860 (+ 2394us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:14:05.581865 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:14:05.581973 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:14:05.582094 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:14:05.584194 (+ 2100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:14:05.584907 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:14:05.587394 (+ 2487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:14:05.587837 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:14:05.587940 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:14:05.592087 (+ 4147us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:14:05.592093 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:14:05.592096 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:14:05.592128 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:14:05.594812 (+ 2684us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:14:05.595548 (+ 736us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:14:05.595558 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:14:05.595560 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:14:05.595674 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:14:05.596100 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:14:05.596104 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:14:05.596107 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:14:05.596368 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:14:05.596522 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0923 20:14:05.597203 (+ 681us) server_negotiation.cc:299] Negotiation successful
0923 20:14:05.597427 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":311,"thread_start_us":193,"threads_started":1}
I20260923 20:14:05.598727 21281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:14:05.578529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.190:43721 (local address 127.1.170.129:40131)
0923 20:14:05.579271 (+ 742us) negotiation.cc:107] Waiting for socket to connect
0923 20:14:05.579323 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0923 20:14:05.581603 (+ 2280us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:14:05.582409 (+ 806us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:14:05.582420 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:14:05.583155 (+ 735us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:14:05.584006 (+ 851us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:14:05.584037 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:14:05.585071 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:14:05.585080 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:14:05.587168 (+ 2088us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:14:05.587186 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:14:05.588079 (+ 893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:14:05.588088 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:14:05.588413 (+ 325us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:14:05.589144 (+ 731us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:14:05.589184 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:14:05.591847 (+ 2663us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:14:05.594964 (+ 3117us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:14:05.594973 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:14:05.594989 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:14:05.595421 (+ 432us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:14:05.595794 (+ 373us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:14:05.595798 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:14:05.595802 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:14:05.595983 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:14:05.596492 (+ 509us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:14:05.596500 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:14:05.596826 (+ 326us) client_negotiation.cc:769] Sending connection context
0923 20:14:05.597085 (+ 259us) client_negotiation.cc:240] Negotiation successful
0923 20:14:05.597604 (+ 519us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":460,"thread_start_us":269,"threads_started":1}
I20260923 20:14:05.600378 21278 heartbeater.cc:347] Connected to a master server at 127.1.170.190:43721
I20260923 20:14:05.600699 21278 heartbeater.cc:464] Registering TS with master...
I20260923 20:14:05.601572 21278 heartbeater.cc:511] Master 127.1.170.190:43721 requested a full tablet report, sending...
I20260923 20:14:05.603768 21089 ts_manager.cc:194] Registered new tserver with Master: 4f69f10b722f451cb008413449c8a1a3 (127.1.170.129:44809)
I20260923 20:14:05.605211 21089 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.170.129@KRBTEST.COM'} at 127.1.170.129:40131
WARNING: no policy specified for kudu/127.1.170.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.170.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.170.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790194121577027-1706-0/krb5kdc/kudu_127.1.170.130.keytab.
Entry for principal kudu/127.1.170.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790194121577027-1706-0/krb5kdc/kudu_127.1.170.130.keytab.
I20260923 20:14:05.637195 1706 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqUuku6/build/asan/bin/kudu
/tmp/dist-test-taskqUuku6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790194121577027-1706-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790194121577027-1706-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790194121577027-1706-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790194121577027-1706-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.170.130:0
--local_ip_for_outbound_sockets=127.1.170.130
--webserver_interface=127.1.170.130
--webserver_port=0
--tserver_master_addrs=127.1.170.190:43721
--keytab_file=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790194121577027-1706-0/krb5kdc/kudu_127.1.170.130.keytab
--principal=kudu/127.1.170.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.170.148:34319
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790194121577027-1706-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790194121577027-1706-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790194121577027-1706-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:14:05.972664 21284 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:14:05.972954 21284 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:14:05.973119 21284 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:14:05.980583 21284 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:14:05.980716 21284 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:14:05.980906 21284 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.170.130
I20260923 20:14:05.991146 21284 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.170.148:34319
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790194121577027-1706-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790194121577027-1706-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790194121577027-1706-0/krb5kdc/kudu_127.1.170.130.keytab
--principal=kudu/127.1.170.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.170.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790194121577027-1706-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.170.130
--webserver_port=0
--tserver_master_addrs=127.1.170.190:43721
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.170.130
--log_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790194121577027-1706-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:14:05.992431 21284 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:14:05.993752 21284 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:14:06.009424 21289 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:14:06.009177 21290 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:14:06.010794 21292 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:14:06.011271 21284 server_base.cc:1034] running on GCE node
I20260923 20:14:06.011911 21284 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:14:06.012928 21284 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:14:06.014159 21284 hybrid_clock.cc:648] HybridClock initialized: now 1790194446014119 us; error 62 us; skew 500 ppm
I20260923 20:14:06.014536 21284 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:14:06 dist-test-slave-th4g krb5kdc[21044](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194446, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.170.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:14:06.017721 21284 init.cc:377] Logged in from keytab as kudu/127.1.170.130@KRBTEST.COM (short username kudu)
I20260923 20:14:06.019429 21284 webserver.cc:460] Webserver started at http://127.1.170.130:46063/ using document root <none> and password file <none>
I20260923 20:14:06.020200 21284 fs_manager.cc:359] Metadata directory not provided
I20260923 20:14:06.020345 21284 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:14:06.020676 21284 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:14:06.023773 21284 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790194121577027-1706-0/minicluster-data/ts-1/data/instance:
uuid: "3d352e4fff3b4d368111a7a7f311c194"
format_stamp: "Formatted at 2026-09-23 20:14:06 on dist-test-slave-th4g"
server_key: "1e39800d2855afe9edbff3dd71a2e243"
server_key_iv: "563fc5d562b10e33df302238fb83d170"
server_key_version: "encryptionkey@0"
I20260923 20:14:06.024557 21284 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790194121577027-1706-0/minicluster-data/ts-1/wal/instance:
uuid: "3d352e4fff3b4d368111a7a7f311c194"
format_stamp: "Formatted at 2026-09-23 20:14:06 on dist-test-slave-th4g"
server_key: "1e39800d2855afe9edbff3dd71a2e243"
server_key_iv: "563fc5d562b10e33df302238fb83d170"
server_key_version: "encryptionkey@0"
I20260923 20:14:06.029644 21284 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260923 20:14:06.033118 21299 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:14:06.034528 21284 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 20:14:06.034706 21284 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790194121577027-1706-0/minicluster-data/ts-1/data,/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790194121577027-1706-0/minicluster-data/ts-1/wal
uuid: "3d352e4fff3b4d368111a7a7f311c194"
format_stamp: "Formatted at 2026-09-23 20:14:06 on dist-test-slave-th4g"
server_key: "1e39800d2855afe9edbff3dd71a2e243"
server_key_iv: "563fc5d562b10e33df302238fb83d170"
server_key_version: "encryptionkey@0"
I20260923 20:14:06.034912 21284 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790194121577027-1706-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790194121577027-1706-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790194121577027-1706-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:14:06.061988 21284 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:14:06.066270 21284 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:14:06.066567 21284 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:14:06.067540 21284 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 20:14:06.069026 21284 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 20:14:06.069126 21284 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 20:14:06.069281 21284 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 20:14:06.069355 21284 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 20:14:06.104018 21284 rpc_server.cc:307] RPC server started. Bound to: 127.1.170.130:37617
I20260923 20:14:06.104106 21411 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.170.130:37617 every 8 connection(s)
I20260923 20:14:06.105347 21284 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790194121577027-1706-0/minicluster-data/ts-1/data/info.pb
I20260923 20:14:06.113309 1706 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqUuku6/build/asan/bin/kudu as pid 21284
I20260923 20:14:06.113481 1706 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790194121577027-1706-0/minicluster-data/ts-1/wal/instance
I20260923 20:14:06.113999 1706 external_mini_cluster.cc:1428] Setting key 3413aa27027f85c3c795d9f75b88c869
Sep 23 20:14:06 dist-test-slave-th4g krb5kdc[21044](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194446, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.170.130@KRBTEST.COM for kudu/127.1.170.190@KRBTEST.COM
I20260923 20:14:06.127276 21414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:14:06.107983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.170.130:44425 (local address 127.1.170.190:43721)
0923 20:14:06.108459 (+ 476us) server_negotiation.cc:206] Beginning negotiation
0923 20:14:06.108466 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:14:06.109396 (+ 930us) server_negotiation.cc:407] Connection header received
0923 20:14:06.111303 (+ 1907us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:14:06.111308 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:14:06.111378 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:14:06.111481 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:14:06.112937 (+ 1456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:14:06.113837 (+ 900us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:14:06.115643 (+ 1806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:14:06.116319 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:14:06.116469 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:14:06.120900 (+ 4431us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:14:06.120906 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:14:06.120910 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:14:06.120951 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:14:06.123902 (+ 2951us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:14:06.124727 (+ 825us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:14:06.124735 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:14:06.124744 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:14:06.124872 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:14:06.125368 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:14:06.125382 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:14:06.125385 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:14:06.125656 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:14:06.125830 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0923 20:14:06.126601 (+ 771us) server_negotiation.cc:299] Negotiation successful
0923 20:14:06.126862 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":349,"thread_start_us":232,"threads_started":1}
I20260923 20:14:06.128330 21415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:14:06.108521 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.190:43721 (local address 127.1.170.130:44425)
0923 20:14:06.109231 (+ 710us) negotiation.cc:107] Waiting for socket to connect
0923 20:14:06.109273 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0923 20:14:06.111030 (+ 1757us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:14:06.111676 (+ 646us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:14:06.111684 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:14:06.112211 (+ 527us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:14:06.112775 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:14:06.112792 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:14:06.114034 (+ 1242us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:14:06.114042 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:14:06.115499 (+ 1457us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:14:06.115511 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:14:06.116600 (+ 1089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:14:06.116608 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:14:06.116896 (+ 288us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:14:06.117681 (+ 785us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:14:06.117716 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:14:06.120615 (+ 2899us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:14:06.124071 (+ 3456us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:14:06.124080 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:14:06.124099 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:14:06.124581 (+ 482us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:14:06.125006 (+ 425us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:14:06.125011 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:14:06.125015 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:14:06.125233 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:14:06.125817 (+ 584us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:14:06.125826 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:14:06.126219 (+ 393us) client_negotiation.cc:769] Sending connection context
0923 20:14:06.126516 (+ 297us) client_negotiation.cc:240] Negotiation successful
0923 20:14:06.127082 (+ 566us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":486,"thread_start_us":280,"threads_started":1}
I20260923 20:14:06.129912 21412 heartbeater.cc:347] Connected to a master server at 127.1.170.190:43721
I20260923 20:14:06.130230 21412 heartbeater.cc:464] Registering TS with master...
I20260923 20:14:06.131067 21412 heartbeater.cc:511] Master 127.1.170.190:43721 requested a full tablet report, sending...
I20260923 20:14:06.132640 21089 ts_manager.cc:194] Registered new tserver with Master: 3d352e4fff3b4d368111a7a7f311c194 (127.1.170.130:37617)
I20260923 20:14:06.133543 21089 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.170.130@KRBTEST.COM'} at 127.1.170.130:44425
WARNING: no policy specified for kudu/127.1.170.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.170.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.170.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790194121577027-1706-0/krb5kdc/kudu_127.1.170.131.keytab.
Entry for principal kudu/127.1.170.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790194121577027-1706-0/krb5kdc/kudu_127.1.170.131.keytab.
I20260923 20:14:06.173254 1706 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqUuku6/build/asan/bin/kudu
/tmp/dist-test-taskqUuku6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790194121577027-1706-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790194121577027-1706-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790194121577027-1706-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790194121577027-1706-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.170.131:0
--local_ip_for_outbound_sockets=127.1.170.131
--webserver_interface=127.1.170.131
--webserver_port=0
--tserver_master_addrs=127.1.170.190:43721
--keytab_file=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790194121577027-1706-0/krb5kdc/kudu_127.1.170.131.keytab
--principal=kudu/127.1.170.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.170.148:34319
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790194121577027-1706-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790194121577027-1706-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790194121577027-1706-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:14:06.499496 21418 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:14:06.499816 21418 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:14:06.499990 21418 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:14:06.507347 21418 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:14:06.507501 21418 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:14:06.507687 21418 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.170.131
I20260923 20:14:06.517594 21418 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.170.148:34319
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790194121577027-1706-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790194121577027-1706-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790194121577027-1706-0/krb5kdc/kudu_127.1.170.131.keytab
--principal=kudu/127.1.170.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.170.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790194121577027-1706-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.170.131
--webserver_port=0
--tserver_master_addrs=127.1.170.190:43721
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.170.131
--log_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790194121577027-1706-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:14:06.518922 21418 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:14:06.520200 21418 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:14:06.533336 21423 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:14:06.533389 21426 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:14:06.533388 21424 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:14:06.534344 21418 server_base.cc:1034] running on GCE node
I20260923 20:14:06.534935 21418 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:14:06.535828 21418 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:14:06.537051 21418 hybrid_clock.cc:648] HybridClock initialized: now 1790194446537002 us; error 40 us; skew 500 ppm
I20260923 20:14:06.537376 21418 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:14:06 dist-test-slave-th4g krb5kdc[21044](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194446, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.170.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:14:06.540719 21418 init.cc:377] Logged in from keytab as kudu/127.1.170.131@KRBTEST.COM (short username kudu)
I20260923 20:14:06.542181 21418 webserver.cc:460] Webserver started at http://127.1.170.131:39391/ using document root <none> and password file <none>
I20260923 20:14:06.542855 21418 fs_manager.cc:359] Metadata directory not provided
I20260923 20:14:06.542997 21418 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:14:06.543311 21418 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:14:06.546188 21418 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790194121577027-1706-0/minicluster-data/ts-2/data/instance:
uuid: "13a9e9a757cc41eeb4ceeb90f08387fd"
format_stamp: "Formatted at 2026-09-23 20:14:06 on dist-test-slave-th4g"
server_key: "06ba338c6369b0d6223992a59bfa93e4"
server_key_iv: "faf49a40b67aa0c69ec7b1f7dbab6f7e"
server_key_version: "encryptionkey@0"
I20260923 20:14:06.547027 21418 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790194121577027-1706-0/minicluster-data/ts-2/wal/instance:
uuid: "13a9e9a757cc41eeb4ceeb90f08387fd"
format_stamp: "Formatted at 2026-09-23 20:14:06 on dist-test-slave-th4g"
server_key: "06ba338c6369b0d6223992a59bfa93e4"
server_key_iv: "faf49a40b67aa0c69ec7b1f7dbab6f7e"
server_key_version: "encryptionkey@0"
I20260923 20:14:06.551950 21418 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260923 20:14:06.555209 21433 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:14:06.556506 21418 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260923 20:14:06.556660 21418 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790194121577027-1706-0/minicluster-data/ts-2/data,/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790194121577027-1706-0/minicluster-data/ts-2/wal
uuid: "13a9e9a757cc41eeb4ceeb90f08387fd"
format_stamp: "Formatted at 2026-09-23 20:14:06 on dist-test-slave-th4g"
server_key: "06ba338c6369b0d6223992a59bfa93e4"
server_key_iv: "faf49a40b67aa0c69ec7b1f7dbab6f7e"
server_key_version: "encryptionkey@0"
I20260923 20:14:06.556823 21418 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790194121577027-1706-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790194121577027-1706-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790194121577027-1706-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:14:06.572878 21418 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:14:06.577090 21418 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:14:06.577456 21418 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:14:06.578517 21418 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 20:14:06.579826 21418 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 20:14:06.579936 21418 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 20:14:06.580089 21418 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 20:14:06.580157 21418 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 20:14:06.608282 21278 heartbeater.cc:503] Master 127.1.170.190:43721 was elected leader, sending a full tablet report...
I20260923 20:14:06.612929 21418 rpc_server.cc:307] RPC server started. Bound to: 127.1.170.131:38231
I20260923 20:14:06.612963 21545 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.170.131:38231 every 8 connection(s)
I20260923 20:14:06.614203 21418 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790194121577027-1706-0/minicluster-data/ts-2/data/info.pb
I20260923 20:14:06.617775 1706 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqUuku6/build/asan/bin/kudu as pid 21418
I20260923 20:14:06.617997 1706 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790194121577027-1706-0/minicluster-data/ts-2/wal/instance
I20260923 20:14:06.618546 1706 external_mini_cluster.cc:1428] Setting key 2c9019a649439afc0813b88fb1d0b9ce
Sep 23 20:14:06 dist-test-slave-th4g krb5kdc[21044](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194446, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.170.131@KRBTEST.COM for kudu/127.1.170.190@KRBTEST.COM
I20260923 20:14:06.633371 21414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:14:06.616621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.170.131:48931 (local address 127.1.170.190:43721)
0923 20:14:06.616794 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0923 20:14:06.616799 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 20:14:06.618095 (+ 1296us) server_negotiation.cc:407] Connection header received
0923 20:14:06.620445 (+ 2350us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:14:06.620452 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:14:06.620536 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:14:06.620632 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:14:06.622232 (+ 1600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:14:06.622954 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:14:06.624367 (+ 1413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:14:06.624730 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:14:06.624823 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:14:06.627980 (+ 3157us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:14:06.627985 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:14:06.627988 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:14:06.628016 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:14:06.630440 (+ 2424us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:14:06.631068 (+ 628us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:14:06.631072 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:14:06.631074 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:14:06.631178 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:14:06.631569 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:14:06.631574 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:14:06.631577 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:14:06.631895 (+ 318us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:14:06.632057 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0923 20:14:06.632813 (+ 756us) server_negotiation.cc:299] Negotiation successful
0923 20:14:06.633044 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260923 20:14:06.634289 21548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:14:06.617073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.190:43721 (local address 127.1.170.131:48931)
0923 20:14:06.617938 (+ 865us) negotiation.cc:107] Waiting for socket to connect
0923 20:14:06.617992 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0923 20:14:06.620128 (+ 2136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:14:06.620821 (+ 693us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:14:06.620829 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:14:06.621511 (+ 682us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:14:06.622087 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:14:06.622105 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:14:06.623078 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:14:06.623082 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:14:06.624245 (+ 1163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:14:06.624256 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:14:06.624904 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:14:06.624909 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:14:06.625096 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:14:06.625679 (+ 583us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:14:06.625700 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:14:06.627793 (+ 2093us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:14:06.630555 (+ 2762us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:14:06.630561 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:14:06.630571 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:14:06.630961 (+ 390us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:14:06.631263 (+ 302us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:14:06.631266 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:14:06.631268 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:14:06.631455 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:14:06.632038 (+ 583us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:14:06.632047 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:14:06.632371 (+ 324us) client_negotiation.cc:769] Sending connection context
0923 20:14:06.632715 (+ 344us) client_negotiation.cc:240] Negotiation successful
0923 20:14:06.633198 (+ 483us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":527,"thread_start_us":307,"threads_started":1}
I20260923 20:14:06.635532 21546 heartbeater.cc:347] Connected to a master server at 127.1.170.190:43721
I20260923 20:14:06.635823 21546 heartbeater.cc:464] Registering TS with master...
I20260923 20:14:06.636485 21546 heartbeater.cc:511] Master 127.1.170.190:43721 requested a full tablet report, sending...
I20260923 20:14:06.637773 21089 ts_manager.cc:194] Registered new tserver with Master: 13a9e9a757cc41eeb4ceeb90f08387fd (127.1.170.131:38231)
I20260923 20:14:06.638602 21089 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.170.131@KRBTEST.COM'} at 127.1.170.131:48931
I20260923 20:14:06.645476 1706 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260923 20:14:06.661445 21414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:14:06.649605 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59700 (local address 127.1.170.190:43721)
0923 20:14:06.649830 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0923 20:14:06.649839 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 20:14:06.650196 (+ 357us) server_negotiation.cc:407] Connection header received
0923 20:14:06.650322 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:14:06.650326 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:14:06.650396 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:14:06.650475 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:14:06.651803 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:14:06.652498 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:14:06.653806 (+ 1308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:14:06.654288 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:14:06.654387 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:14:06.656598 (+ 2211us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:14:06.656603 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:14:06.656606 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:14:06.656635 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:14:06.659142 (+ 2507us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:14:06.659788 (+ 646us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:14:06.659792 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:14:06.659794 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:14:06.659915 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:14:06.660226 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:14:06.660229 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:14:06.660231 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:14:06.660537 (+ 306us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:14:06.660731 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0923 20:14:06.660996 (+ 265us) server_negotiation.cc:299] Negotiation successful
0923 20:14:06.661142 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260923 20:14:06.665282 21089 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59700:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 20:14:06.672083 21089 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 20:14:06.705945 21562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:14:06.694283 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60580 (local address 127.1.170.131:38231)
0923 20:14:06.694982 (+ 699us) server_negotiation.cc:206] Beginning negotiation
0923 20:14:06.694991 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 20:14:06.697906 (+ 2915us) server_negotiation.cc:407] Connection header received
0923 20:14:06.698255 (+ 349us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:14:06.698268 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:14:06.698552 (+ 284us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:14:06.698760 (+ 208us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:14:06.699606 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:14:06.701591 (+ 1985us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:14:06.703569 (+ 1978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:14:06.704740 (+ 1171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:14:06.704868 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:14:06.705100 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0923 20:14:06.705373 (+ 273us) server_negotiation.cc:299] Negotiation successful
0923 20:14:06.705542 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":545,"thread_start_us":398,"threads_started":1}
I20260923 20:14:06.706460 21563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:14:06.694990 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.131:38231 (local address 127.0.0.1:60580)
0923 20:14:06.697765 (+ 2775us) negotiation.cc:107] Waiting for socket to connect
0923 20:14:06.697785 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0923 20:14:06.698019 (+ 234us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:14:06.698769 (+ 750us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:14:06.698776 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:14:06.698806 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:14:06.699345 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:14:06.699369 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:14:06.701725 (+ 2356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:14:06.701730 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:14:06.703410 (+ 1680us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:14:06.703422 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:14:06.704933 (+ 1511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:14:06.704939 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:14:06.705127 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:14:06.705178 (+ 51us) client_negotiation.cc:769] Sending connection context
0923 20:14:06.705315 (+ 137us) client_negotiation.cc:240] Negotiation successful
0923 20:14:06.705504 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2654,"thread_start_us":2519,"threads_started":1}
I20260923 20:14:06.706558 21558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:14:06.683771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.130:37617 (local address 127.0.0.1:36232)
0923 20:14:06.693845 (+ 10074us) negotiation.cc:107] Waiting for socket to connect
0923 20:14:06.693863 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0923 20:14:06.693981 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:14:06.699074 (+ 5093us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:14:06.699078 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:14:06.699102 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:14:06.699606 (+ 504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:14:06.699622 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:14:06.701349 (+ 1727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:14:06.701354 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:14:06.704112 (+ 2758us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:14:06.704126 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:14:06.705685 (+ 1559us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:14:06.705691 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:14:06.705845 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:14:06.705920 (+ 75us) client_negotiation.cc:769] Sending connection context
0923 20:14:06.705984 (+ 64us) client_negotiation.cc:240] Negotiation successful
0923 20:14:06.706113 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9946,"mutex_wait_us":107,"thread_start_us":9042,"threads_started":1}
I20260923 20:14:06.707193 21559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:14:06.683510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.129:44809 (local address 127.0.0.1:34014)
0923 20:14:06.693213 (+ 9703us) negotiation.cc:107] Waiting for socket to connect
0923 20:14:06.693249 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0923 20:14:06.693424 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:14:06.698791 (+ 5367us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:14:06.698795 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:14:06.698821 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:14:06.699433 (+ 612us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:14:06.699447 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:14:06.701551 (+ 2104us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:14:06.701558 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:14:06.704119 (+ 2561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:14:06.704126 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:14:06.706192 (+ 2066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:14:06.706198 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:14:06.706377 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:14:06.706411 (+ 34us) client_negotiation.cc:769] Sending connection context
0923 20:14:06.706508 (+ 97us) client_negotiation.cc:240] Negotiation successful
0923 20:14:06.706632 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9458,"mutex_wait_us":64,"thread_start_us":10071,"threads_started":1}
I20260923 20:14:06.707407 21561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:14:06.684337 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36232 (local address 127.1.170.130:37617)
0923 20:14:06.698555 (+ 14218us) server_negotiation.cc:206] Beginning negotiation
0923 20:14:06.698562 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:14:06.698589 (+ 27us) server_negotiation.cc:407] Connection header received
0923 20:14:06.698682 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:14:06.698692 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:14:06.698896 (+ 204us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:14:06.699083 (+ 187us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:14:06.699961 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:14:06.701208 (+ 1247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:14:06.704318 (+ 3110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:14:06.705469 (+ 1151us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:14:06.706693 (+ 1224us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:14:06.706886 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0923 20:14:06.706949 (+ 63us) server_negotiation.cc:299] Negotiation successful
0923 20:14:06.707104 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14073,"thread_start_us":13961,"threads_started":1}
I20260923 20:14:06.707983 21560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:14:06.683828 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34014 (local address 127.1.170.129:44809)
0923 20:14:06.698189 (+ 14361us) server_negotiation.cc:206] Beginning negotiation
0923 20:14:06.698198 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 20:14:06.698228 (+ 30us) server_negotiation.cc:407] Connection header received
0923 20:14:06.698327 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:14:06.698337 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:14:06.698567 (+ 230us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:14:06.699637 (+ 1070us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:14:06.699885 (+ 248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:14:06.701413 (+ 1528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:14:06.704297 (+ 2884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:14:06.705646 (+ 1349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:14:06.705791 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:14:06.706031 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0923 20:14:06.707512 (+ 1481us) server_negotiation.cc:299] Negotiation successful
0923 20:14:06.707660 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14185,"thread_start_us":14040,"threads_started":1}
I20260923 20:14:06.710561 21481 tablet_service.cc:1467] Processing CreateTablet for tablet 49f7da30562f426ab0b407212e173fa3 (DEFAULT_TABLE table=test-table [id=44df16e769d74cafb24bd8eb4271b851]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 20:14:06.710772 21347 tablet_service.cc:1467] Processing CreateTablet for tablet 49f7da30562f426ab0b407212e173fa3 (DEFAULT_TABLE table=test-table [id=44df16e769d74cafb24bd8eb4271b851]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 20:14:06.711303 21213 tablet_service.cc:1467] Processing CreateTablet for tablet 49f7da30562f426ab0b407212e173fa3 (DEFAULT_TABLE table=test-table [id=44df16e769d74cafb24bd8eb4271b851]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 20:14:06.711792 21481 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 49f7da30562f426ab0b407212e173fa3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:14:06.712082 21347 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 49f7da30562f426ab0b407212e173fa3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:14:06.712548 21213 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 49f7da30562f426ab0b407212e173fa3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:14:06.723351 21564 tablet_bootstrap.cc:492] T 49f7da30562f426ab0b407212e173fa3 P 13a9e9a757cc41eeb4ceeb90f08387fd: Bootstrap starting.
I20260923 20:14:06.726581 21564 tablet_bootstrap.cc:654] T 49f7da30562f426ab0b407212e173fa3 P 13a9e9a757cc41eeb4ceeb90f08387fd: Neither blocks nor log segments found. Creating new log.
I20260923 20:14:06.726956 21565 tablet_bootstrap.cc:492] T 49f7da30562f426ab0b407212e173fa3 P 4f69f10b722f451cb008413449c8a1a3: Bootstrap starting.
I20260923 20:14:06.728022 21564 log.cc:824] T 49f7da30562f426ab0b407212e173fa3 P 13a9e9a757cc41eeb4ceeb90f08387fd: Log is configured to *not* fsync() on all Append() calls
I20260923 20:14:06.728497 21566 tablet_bootstrap.cc:492] T 49f7da30562f426ab0b407212e173fa3 P 3d352e4fff3b4d368111a7a7f311c194: Bootstrap starting.
I20260923 20:14:06.730729 21565 tablet_bootstrap.cc:654] T 49f7da30562f426ab0b407212e173fa3 P 4f69f10b722f451cb008413449c8a1a3: Neither blocks nor log segments found. Creating new log.
I20260923 20:14:06.731477 21566 tablet_bootstrap.cc:654] T 49f7da30562f426ab0b407212e173fa3 P 3d352e4fff3b4d368111a7a7f311c194: Neither blocks nor log segments found. Creating new log.
I20260923 20:14:06.731940 21565 log.cc:824] T 49f7da30562f426ab0b407212e173fa3 P 4f69f10b722f451cb008413449c8a1a3: Log is configured to *not* fsync() on all Append() calls
I20260923 20:14:06.732451 21566 log.cc:824] T 49f7da30562f426ab0b407212e173fa3 P 3d352e4fff3b4d368111a7a7f311c194: Log is configured to *not* fsync() on all Append() calls
I20260923 20:14:06.732607 21564 tablet_bootstrap.cc:492] T 49f7da30562f426ab0b407212e173fa3 P 13a9e9a757cc41eeb4ceeb90f08387fd: No bootstrap required, opened a new log
I20260923 20:14:06.732877 21564 ts_tablet_manager.cc:1397] T 49f7da30562f426ab0b407212e173fa3 P 13a9e9a757cc41eeb4ceeb90f08387fd: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.002s
I20260923 20:14:06.735350 21565 tablet_bootstrap.cc:492] T 49f7da30562f426ab0b407212e173fa3 P 4f69f10b722f451cb008413449c8a1a3: No bootstrap required, opened a new log
I20260923 20:14:06.735630 21565 ts_tablet_manager.cc:1397] T 49f7da30562f426ab0b407212e173fa3 P 4f69f10b722f451cb008413449c8a1a3: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.004s
I20260923 20:14:06.737417 21566 tablet_bootstrap.cc:492] T 49f7da30562f426ab0b407212e173fa3 P 3d352e4fff3b4d368111a7a7f311c194: No bootstrap required, opened a new log
I20260923 20:14:06.737668 21566 ts_tablet_manager.cc:1397] T 49f7da30562f426ab0b407212e173fa3 P 3d352e4fff3b4d368111a7a7f311c194: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.008s
I20260923 20:14:06.743475 21564 raft_consensus.cc:348] T 49f7da30562f426ab0b407212e173fa3 P 13a9e9a757cc41eeb4ceeb90f08387fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f69f10b722f451cb008413449c8a1a3" member_type: VOTER last_known_addr { host: "127.1.170.129" port: 44809 } } peers { permanent_uuid: "3d352e4fff3b4d368111a7a7f311c194" member_type: VOTER last_known_addr { host: "127.1.170.130" port: 37617 } } peers { permanent_uuid: "13a9e9a757cc41eeb4ceeb90f08387fd" member_type: VOTER last_known_addr { host: "127.1.170.131" port: 38231 } }
I20260923 20:14:06.743953 21564 raft_consensus.cc:374] T 49f7da30562f426ab0b407212e173fa3 P 13a9e9a757cc41eeb4ceeb90f08387fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:14:06.744082 21564 raft_consensus.cc:729] T 49f7da30562f426ab0b407212e173fa3 P 13a9e9a757cc41eeb4ceeb90f08387fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 13a9e9a757cc41eeb4ceeb90f08387fd, State: Initialized, Role: FOLLOWER
I20260923 20:14:06.744722 21564 consensus_queue.cc:260] T 49f7da30562f426ab0b407212e173fa3 P 13a9e9a757cc41eeb4ceeb90f08387fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f69f10b722f451cb008413449c8a1a3" member_type: VOTER last_known_addr { host: "127.1.170.129" port: 44809 } } peers { permanent_uuid: "3d352e4fff3b4d368111a7a7f311c194" member_type: VOTER last_known_addr { host: "127.1.170.130" port: 37617 } } peers { permanent_uuid: "13a9e9a757cc41eeb4ceeb90f08387fd" member_type: VOTER last_known_addr { host: "127.1.170.131" port: 38231 } }
I20260923 20:14:06.745072 21566 raft_consensus.cc:348] T 49f7da30562f426ab0b407212e173fa3 P 3d352e4fff3b4d368111a7a7f311c194 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f69f10b722f451cb008413449c8a1a3" member_type: VOTER last_known_addr { host: "127.1.170.129" port: 44809 } } peers { permanent_uuid: "3d352e4fff3b4d368111a7a7f311c194" member_type: VOTER last_known_addr { host: "127.1.170.130" port: 37617 } } peers { permanent_uuid: "13a9e9a757cc41eeb4ceeb90f08387fd" member_type: VOTER last_known_addr { host: "127.1.170.131" port: 38231 } }
I20260923 20:14:06.745568 21566 raft_consensus.cc:374] T 49f7da30562f426ab0b407212e173fa3 P 3d352e4fff3b4d368111a7a7f311c194 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:14:06.745693 21566 raft_consensus.cc:729] T 49f7da30562f426ab0b407212e173fa3 P 3d352e4fff3b4d368111a7a7f311c194 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d352e4fff3b4d368111a7a7f311c194, State: Initialized, Role: FOLLOWER
I20260923 20:14:06.745998 21546 heartbeater.cc:503] Master 127.1.170.190:43721 was elected leader, sending a full tablet report...
I20260923 20:14:06.746292 21566 consensus_queue.cc:260] T 49f7da30562f426ab0b407212e173fa3 P 3d352e4fff3b4d368111a7a7f311c194 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f69f10b722f451cb008413449c8a1a3" member_type: VOTER last_known_addr { host: "127.1.170.129" port: 44809 } } peers { permanent_uuid: "3d352e4fff3b4d368111a7a7f311c194" member_type: VOTER last_known_addr { host: "127.1.170.130" port: 37617 } } peers { permanent_uuid: "13a9e9a757cc41eeb4ceeb90f08387fd" member_type: VOTER last_known_addr { host: "127.1.170.131" port: 38231 } }
I20260923 20:14:06.746210 21565 raft_consensus.cc:348] T 49f7da30562f426ab0b407212e173fa3 P 4f69f10b722f451cb008413449c8a1a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f69f10b722f451cb008413449c8a1a3" member_type: VOTER last_known_addr { host: "127.1.170.129" port: 44809 } } peers { permanent_uuid: "3d352e4fff3b4d368111a7a7f311c194" member_type: VOTER last_known_addr { host: "127.1.170.130" port: 37617 } } peers { permanent_uuid: "13a9e9a757cc41eeb4ceeb90f08387fd" member_type: VOTER last_known_addr { host: "127.1.170.131" port: 38231 } }
I20260923 20:14:06.746606 21565 raft_consensus.cc:374] T 49f7da30562f426ab0b407212e173fa3 P 4f69f10b722f451cb008413449c8a1a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:14:06.746701 21565 raft_consensus.cc:729] T 49f7da30562f426ab0b407212e173fa3 P 4f69f10b722f451cb008413449c8a1a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4f69f10b722f451cb008413449c8a1a3, State: Initialized, Role: FOLLOWER
I20260923 20:14:06.747380 21565 consensus_queue.cc:260] T 49f7da30562f426ab0b407212e173fa3 P 4f69f10b722f451cb008413449c8a1a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f69f10b722f451cb008413449c8a1a3" member_type: VOTER last_known_addr { host: "127.1.170.129" port: 44809 } } peers { permanent_uuid: "3d352e4fff3b4d368111a7a7f311c194" member_type: VOTER last_known_addr { host: "127.1.170.130" port: 37617 } } peers { permanent_uuid: "13a9e9a757cc41eeb4ceeb90f08387fd" member_type: VOTER last_known_addr { host: "127.1.170.131" port: 38231 } }
I20260923 20:14:06.748554 21412 heartbeater.cc:503] Master 127.1.170.190:43721 was elected leader, sending a full tablet report...
I20260923 20:14:06.749004 21566 ts_tablet_manager.cc:1428] T 49f7da30562f426ab0b407212e173fa3 P 3d352e4fff3b4d368111a7a7f311c194: Time spent starting tablet: real 0.011s user 0.007s sys 0.004s
I20260923 20:14:06.750676 21564 ts_tablet_manager.cc:1428] T 49f7da30562f426ab0b407212e173fa3 P 13a9e9a757cc41eeb4ceeb90f08387fd: Time spent starting tablet: real 0.018s user 0.008s sys 0.005s
I20260923 20:14:06.752053 21565 ts_tablet_manager.cc:1428] T 49f7da30562f426ab0b407212e173fa3 P 4f69f10b722f451cb008413449c8a1a3: Time spent starting tablet: real 0.016s user 0.008s sys 0.006s
I20260923 20:14:06.770256 21570 raft_consensus.cc:482] T 49f7da30562f426ab0b407212e173fa3 P 13a9e9a757cc41eeb4ceeb90f08387fd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 20:14:06.770507 21570 raft_consensus.cc:504] T 49f7da30562f426ab0b407212e173fa3 P 13a9e9a757cc41eeb4ceeb90f08387fd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f69f10b722f451cb008413449c8a1a3" member_type: VOTER last_known_addr { host: "127.1.170.129" port: 44809 } } peers { permanent_uuid: "3d352e4fff3b4d368111a7a7f311c194" member_type: VOTER last_known_addr { host: "127.1.170.130" port: 37617 } } peers { permanent_uuid: "13a9e9a757cc41eeb4ceeb90f08387fd" member_type: VOTER last_known_addr { host: "127.1.170.131" port: 38231 } }
I20260923 20:14:06.772217 21570 leader_election.cc:288] T 49f7da30562f426ab0b407212e173fa3 P 13a9e9a757cc41eeb4ceeb90f08387fd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4f69f10b722f451cb008413449c8a1a3 (127.1.170.129:44809), 3d352e4fff3b4d368111a7a7f311c194 (127.1.170.130:37617)
I20260923 20:14:06.777767 21561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:14:06.772837 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.170.131:46473 (local address 127.1.170.130:37617)
0923 20:14:06.773013 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0923 20:14:06.773018 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 20:14:06.773033 (+ 15us) server_negotiation.cc:407] Connection header received
0923 20:14:06.773089 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:14:06.773093 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:14:06.773148 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:14:06.773259 (+ 111us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:14:06.773676 (+ 417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:14:06.774419 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:14:06.776076 (+ 1657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:14:06.776870 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:14:06.777027 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:14:06.777099 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0923 20:14:06.777380 (+ 281us) server_negotiation.cc:299] Negotiation successful
0923 20:14:06.777508 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260923 20:14:06.777765 21548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:14:06.772531 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.130:37617 (local address 127.1.170.131:46473)
0923 20:14:06.772726 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0923 20:14:06.772750 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0923 20:14:06.772888 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:14:06.773279 (+ 391us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:14:06.773283 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:14:06.773310 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:14:06.773522 (+ 212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:14:06.773531 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:14:06.774568 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:14:06.774573 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:14:06.775917 (+ 1344us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:14:06.775929 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:14:06.777018 (+ 1089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:14:06.777023 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:14:06.777200 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:14:06.777222 (+ 22us) client_negotiation.cc:769] Sending connection context
0923 20:14:06.777352 (+ 130us) client_negotiation.cc:240] Negotiation successful
0923 20:14:06.777500 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39}
I20260923 20:14:06.778878 21573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:14:06.772554 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.129:44809 (local address 127.1.170.131:46219)
0923 20:14:06.773694 (+ 1140us) negotiation.cc:107] Waiting for socket to connect
0923 20:14:06.773707 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0923 20:14:06.773838 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:14:06.774166 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:14:06.774170 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:14:06.774192 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:14:06.774499 (+ 307us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:14:06.774507 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:14:06.775571 (+ 1064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:14:06.775577 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:14:06.776884 (+ 1307us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:14:06.776895 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:14:06.778264 (+ 1369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:14:06.778273 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:14:06.778432 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:14:06.778455 (+ 23us) client_negotiation.cc:769] Sending connection context
0923 20:14:06.778529 (+ 74us) client_negotiation.cc:240] Negotiation successful
0923 20:14:06.778628 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1047,"mutex_wait_us":429,"thread_start_us":387,"threads_started":1}
I20260923 20:14:06.778959 21560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:14:06.772636 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.170.131:46219 (local address 127.1.170.129:44809)
0923 20:14:06.772823 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0923 20:14:06.772827 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 20:14:06.773797 (+ 970us) server_negotiation.cc:407] Connection header received
0923 20:14:06.773996 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:14:06.774000 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:14:06.774060 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:14:06.774153 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:14:06.774648 (+ 495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:14:06.775433 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:14:06.777021 (+ 1588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:14:06.778080 (+ 1059us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:14:06.778221 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:14:06.778268 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0923 20:14:06.778586 (+ 318us) server_negotiation.cc:299] Negotiation successful
0923 20:14:06.778712 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260923 20:14:06.778873 21367 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49f7da30562f426ab0b407212e173fa3" candidate_uuid: "13a9e9a757cc41eeb4ceeb90f08387fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d352e4fff3b4d368111a7a7f311c194" is_pre_election: true
I20260923 20:14:06.779384 21367 raft_consensus.cc:2454] T 49f7da30562f426ab0b407212e173fa3 P 3d352e4fff3b4d368111a7a7f311c194 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 13a9e9a757cc41eeb4ceeb90f08387fd in term 0.
I20260923 20:14:06.780035 21435 leader_election.cc:302] T 49f7da30562f426ab0b407212e173fa3 P 13a9e9a757cc41eeb4ceeb90f08387fd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 13a9e9a757cc41eeb4ceeb90f08387fd, 3d352e4fff3b4d368111a7a7f311c194; no voters:
I20260923 20:14:06.780205 21233 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49f7da30562f426ab0b407212e173fa3" candidate_uuid: "13a9e9a757cc41eeb4ceeb90f08387fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4f69f10b722f451cb008413449c8a1a3" is_pre_election: true
I20260923 20:14:06.780419 21570 raft_consensus.cc:2781] T 49f7da30562f426ab0b407212e173fa3 P 13a9e9a757cc41eeb4ceeb90f08387fd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 20:14:06.780563 21570 raft_consensus.cc:482] T 49f7da30562f426ab0b407212e173fa3 P 13a9e9a757cc41eeb4ceeb90f08387fd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 20:14:06.780676 21570 raft_consensus.cc:3037] T 49f7da30562f426ab0b407212e173fa3 P 13a9e9a757cc41eeb4ceeb90f08387fd [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:14:06.780767 21233 raft_consensus.cc:2454] T 49f7da30562f426ab0b407212e173fa3 P 4f69f10b722f451cb008413449c8a1a3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 13a9e9a757cc41eeb4ceeb90f08387fd in term 0.
I20260923 20:14:06.782990 21570 raft_consensus.cc:504] T 49f7da30562f426ab0b407212e173fa3 P 13a9e9a757cc41eeb4ceeb90f08387fd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f69f10b722f451cb008413449c8a1a3" member_type: VOTER last_known_addr { host: "127.1.170.129" port: 44809 } } peers { permanent_uuid: "3d352e4fff3b4d368111a7a7f311c194" member_type: VOTER last_known_addr { host: "127.1.170.130" port: 37617 } } peers { permanent_uuid: "13a9e9a757cc41eeb4ceeb90f08387fd" member_type: VOTER last_known_addr { host: "127.1.170.131" port: 38231 } }
I20260923 20:14:06.783514 21570 leader_election.cc:288] T 49f7da30562f426ab0b407212e173fa3 P 13a9e9a757cc41eeb4ceeb90f08387fd [CANDIDATE]: Term 1 election: Requested vote from peers 4f69f10b722f451cb008413449c8a1a3 (127.1.170.129:44809), 3d352e4fff3b4d368111a7a7f311c194 (127.1.170.130:37617)
I20260923 20:14:06.783916 21233 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49f7da30562f426ab0b407212e173fa3" candidate_uuid: "13a9e9a757cc41eeb4ceeb90f08387fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4f69f10b722f451cb008413449c8a1a3"
I20260923 20:14:06.783964 21367 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49f7da30562f426ab0b407212e173fa3" candidate_uuid: "13a9e9a757cc41eeb4ceeb90f08387fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d352e4fff3b4d368111a7a7f311c194"
I20260923 20:14:06.784111 21233 raft_consensus.cc:3037] T 49f7da30562f426ab0b407212e173fa3 P 4f69f10b722f451cb008413449c8a1a3 [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:14:06.784159 21367 raft_consensus.cc:3037] T 49f7da30562f426ab0b407212e173fa3 P 3d352e4fff3b4d368111a7a7f311c194 [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:14:06.786317 21233 raft_consensus.cc:2454] T 49f7da30562f426ab0b407212e173fa3 P 4f69f10b722f451cb008413449c8a1a3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 13a9e9a757cc41eeb4ceeb90f08387fd in term 1.
I20260923 20:14:06.786458 21367 raft_consensus.cc:2454] T 49f7da30562f426ab0b407212e173fa3 P 3d352e4fff3b4d368111a7a7f311c194 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 13a9e9a757cc41eeb4ceeb90f08387fd in term 1.
I20260923 20:14:06.786744 21436 leader_election.cc:302] T 49f7da30562f426ab0b407212e173fa3 P 13a9e9a757cc41eeb4ceeb90f08387fd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 13a9e9a757cc41eeb4ceeb90f08387fd, 4f69f10b722f451cb008413449c8a1a3; no voters:
I20260923 20:14:06.787034 21570 raft_consensus.cc:2781] T 49f7da30562f426ab0b407212e173fa3 P 13a9e9a757cc41eeb4ceeb90f08387fd [term 1 FOLLOWER]: Leader election won for term 1
I20260923 20:14:06.787398 21570 raft_consensus.cc:686] T 49f7da30562f426ab0b407212e173fa3 P 13a9e9a757cc41eeb4ceeb90f08387fd [term 1 LEADER]: Becoming Leader. State: Replica: 13a9e9a757cc41eeb4ceeb90f08387fd, State: Running, Role: LEADER
I20260923 20:14:06.787823 21570 consensus_queue.cc:237] T 49f7da30562f426ab0b407212e173fa3 P 13a9e9a757cc41eeb4ceeb90f08387fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f69f10b722f451cb008413449c8a1a3" member_type: VOTER last_known_addr { host: "127.1.170.129" port: 44809 } } peers { permanent_uuid: "3d352e4fff3b4d368111a7a7f311c194" member_type: VOTER last_known_addr { host: "127.1.170.130" port: 37617 } } peers { permanent_uuid: "13a9e9a757cc41eeb4ceeb90f08387fd" member_type: VOTER last_known_addr { host: "127.1.170.131" port: 38231 } }
I20260923 20:14:06.792338 21088 catalog_manager.cc:5495] T 49f7da30562f426ab0b407212e173fa3 P 13a9e9a757cc41eeb4ceeb90f08387fd reported cstate change: term changed from 0 to 1, leader changed from <none> to 13a9e9a757cc41eeb4ceeb90f08387fd (127.1.170.131). New cstate: current_term: 1 leader_uuid: "13a9e9a757cc41eeb4ceeb90f08387fd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f69f10b722f451cb008413449c8a1a3" member_type: VOTER last_known_addr { host: "127.1.170.129" port: 44809 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3d352e4fff3b4d368111a7a7f311c194" member_type: VOTER last_known_addr { host: "127.1.170.130" port: 37617 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "13a9e9a757cc41eeb4ceeb90f08387fd" member_type: VOTER last_known_addr { host: "127.1.170.131" port: 38231 } health_report { overall_health: HEALTHY } } }
I20260923 20:14:06.818885 21562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:14:06.813456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60590 (local address 127.1.170.131:38231)
0923 20:14:06.813689 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0923 20:14:06.813694 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 20:14:06.813725 (+ 31us) server_negotiation.cc:407] Connection header received
0923 20:14:06.814021 (+ 296us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:14:06.814025 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:14:06.814102 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:14:06.814247 (+ 145us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 20:14:06.814586 (+ 339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:14:06.815358 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:14:06.816732 (+ 1374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:14:06.817147 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:14:06.817263 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:14:06.817714 (+ 451us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 20:14:06.818263 (+ 549us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 20:14:06.818380 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0923 20:14:06.818506 (+ 126us) server_negotiation.cc:299] Negotiation successful
0923 20:14:06.818573 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260923 20:14:06.826663 21367 raft_consensus.cc:1261] T 49f7da30562f426ab0b407212e173fa3 P 3d352e4fff3b4d368111a7a7f311c194 [term 1 FOLLOWER]: Refusing update from remote peer 13a9e9a757cc41eeb4ceeb90f08387fd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 20:14:06.826669 21233 raft_consensus.cc:1261] T 49f7da30562f426ab0b407212e173fa3 P 4f69f10b722f451cb008413449c8a1a3 [term 1 FOLLOWER]: Refusing update from remote peer 13a9e9a757cc41eeb4ceeb90f08387fd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 20:14:06.827605 21574 consensus_queue.cc:1035] T 49f7da30562f426ab0b407212e173fa3 P 13a9e9a757cc41eeb4ceeb90f08387fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "4f69f10b722f451cb008413449c8a1a3" member_type: VOTER last_known_addr { host: "127.1.170.129" port: 44809 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260923 20:14:06.827839 21279 tablet.cc:2369] T 49f7da30562f426ab0b407212e173fa3 P 4f69f10b722f451cb008413449c8a1a3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 20:14:06.828087 21570 consensus_queue.cc:1035] T 49f7da30562f426ab0b407212e173fa3 P 13a9e9a757cc41eeb4ceeb90f08387fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d352e4fff3b4d368111a7a7f311c194" member_type: VOTER last_known_addr { host: "127.1.170.130" port: 37617 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 20:14:06.852257 21089 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59700:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 20:14:06.852566 21089 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59700:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 20:14:06.857790 21089 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4f63fafbc48f4491848bc1a9e15565e1: Sending DeleteTablet for 3 replicas of tablet 49f7da30562f426ab0b407212e173fa3
I20260923 20:14:06.859053 21213 tablet_service.cc:1514] Processing DeleteTablet for tablet 49f7da30562f426ab0b407212e173fa3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 20:14:06 UTC) from {username='kudu', principal='kudu/127.1.170.190@KRBTEST.COM'} at 127.0.0.1:34014
I20260923 20:14:06.859732 21481 tablet_service.cc:1514] Processing DeleteTablet for tablet 49f7da30562f426ab0b407212e173fa3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 20:14:06 UTC) from {username='kudu', principal='kudu/127.1.170.190@KRBTEST.COM'} at 127.0.0.1:60580
I20260923 20:14:06.860261 21347 tablet_service.cc:1514] Processing DeleteTablet for tablet 49f7da30562f426ab0b407212e173fa3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 20:14:06 UTC) from {username='kudu', principal='kudu/127.1.170.190@KRBTEST.COM'} at 127.0.0.1:36232
I20260923 20:14:06.864550 1706 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskqUuku6/build/asan/bin/kudu pid 21150
I20260923 20:14:06.874977 21588 tablet_replica.cc:330] T 49f7da30562f426ab0b407212e173fa3 P 13a9e9a757cc41eeb4ceeb90f08387fd: stopping tablet replica
I20260923 20:14:06.875471 21588 raft_consensus.cc:2229] T 49f7da30562f426ab0b407212e173fa3 P 13a9e9a757cc41eeb4ceeb90f08387fd [term 1 LEADER]: Raft consensus shutting down.
I20260923 20:14:06.876286 21588 raft_consensus.cc:2258] T 49f7da30562f426ab0b407212e173fa3 P 13a9e9a757cc41eeb4ceeb90f08387fd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 20:14:06.878716 21588 ts_tablet_manager.cc:1905] T 49f7da30562f426ab0b407212e173fa3 P 13a9e9a757cc41eeb4ceeb90f08387fd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 20:14:06.886953 21588 ts_tablet_manager.cc:1918] T 49f7da30562f426ab0b407212e173fa3 P 13a9e9a757cc41eeb4ceeb90f08387fd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 20:14:06.887147 21588 log.cc:1196] T 49f7da30562f426ab0b407212e173fa3 P 13a9e9a757cc41eeb4ceeb90f08387fd: Deleting WAL directory at /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790194121577027-1706-0/minicluster-data/ts-2/wal/wals/49f7da30562f426ab0b407212e173fa3
I20260923 20:14:06.887677 21588 ts_tablet_manager.cc:1939] T 49f7da30562f426ab0b407212e173fa3 P 13a9e9a757cc41eeb4ceeb90f08387fd: Deleting consensus metadata
I20260923 20:14:06.888301 21589 tablet_replica.cc:330] T 49f7da30562f426ab0b407212e173fa3 P 3d352e4fff3b4d368111a7a7f311c194: stopping tablet replica
I20260923 20:14:06.888775 21589 raft_consensus.cc:2229] T 49f7da30562f426ab0b407212e173fa3 P 3d352e4fff3b4d368111a7a7f311c194 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 20:14:06.889294 21589 raft_consensus.cc:2258] T 49f7da30562f426ab0b407212e173fa3 P 3d352e4fff3b4d368111a7a7f311c194 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 20:14:06.889580 21075 catalog_manager.cc:4841] TS 13a9e9a757cc41eeb4ceeb90f08387fd (127.1.170.131:38231): tablet 49f7da30562f426ab0b407212e173fa3 (table test-table [id=44df16e769d74cafb24bd8eb4271b851]) successfully deleted
I20260923 20:14:06.890120 21587 tablet_replica.cc:330] T 49f7da30562f426ab0b407212e173fa3 P 4f69f10b722f451cb008413449c8a1a3: stopping tablet replica
I20260923 20:14:06.892619 21589 ts_tablet_manager.cc:1905] T 49f7da30562f426ab0b407212e173fa3 P 3d352e4fff3b4d368111a7a7f311c194: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 20:14:06.893966 21587 raft_consensus.cc:2229] T 49f7da30562f426ab0b407212e173fa3 P 4f69f10b722f451cb008413449c8a1a3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 20:14:06.894498 21587 raft_consensus.cc:2258] T 49f7da30562f426ab0b407212e173fa3 P 4f69f10b722f451cb008413449c8a1a3 [term 1 FOLLOWER]: Raft consensus is shut down!
Sep 23 20:14:06 dist-test-slave-th4g krb5kdc[21044](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194444, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.170.129@KRBTEST.COM
I20260923 20:14:06.896548 21587 ts_tablet_manager.cc:1905] T 49f7da30562f426ab0b407212e173fa3 P 4f69f10b722f451cb008413449c8a1a3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 20:14:06.901099 21589 ts_tablet_manager.cc:1918] T 49f7da30562f426ab0b407212e173fa3 P 3d352e4fff3b4d368111a7a7f311c194: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 20:14:06.901257 21589 log.cc:1196] T 49f7da30562f426ab0b407212e173fa3 P 3d352e4fff3b4d368111a7a7f311c194: Deleting WAL directory at /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790194121577027-1706-0/minicluster-data/ts-1/wal/wals/49f7da30562f426ab0b407212e173fa3
I20260923 20:14:06.901803 21589 ts_tablet_manager.cc:1939] T 49f7da30562f426ab0b407212e173fa3 P 3d352e4fff3b4d368111a7a7f311c194: Deleting consensus metadata
I20260923 20:14:06.903620 21587 ts_tablet_manager.cc:1918] T 49f7da30562f426ab0b407212e173fa3 P 4f69f10b722f451cb008413449c8a1a3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 20:14:06.903638 21074 catalog_manager.cc:4841] TS 3d352e4fff3b4d368111a7a7f311c194 (127.1.170.130:37617): tablet 49f7da30562f426ab0b407212e173fa3 (table test-table [id=44df16e769d74cafb24bd8eb4271b851]) successfully deleted
I20260923 20:14:06.903789 21587 log.cc:1196] T 49f7da30562f426ab0b407212e173fa3 P 4f69f10b722f451cb008413449c8a1a3: Deleting WAL directory at /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790194121577027-1706-0/minicluster-data/ts-0/wal/wals/49f7da30562f426ab0b407212e173fa3
I20260923 20:14:06.903817 21560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:14:06.869453 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34018 (local address 127.1.170.129:44809)
0923 20:14:06.869638 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0923 20:14:06.869644 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:14:06.881906 (+ 12262us) server_negotiation.cc:407] Connection header received
0923 20:14:06.883123 (+ 1217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:14:06.883132 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:14:06.883239 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:14:06.883365 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:14:06.885157 (+ 1792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:14:06.886414 (+ 1257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:14:06.888692 (+ 2278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:14:06.889333 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:14:06.889424 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:14:06.895985 (+ 6561us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:14:06.896000 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:14:06.896020 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:14:06.896064 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:14:06.899921 (+ 3857us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:14:06.900731 (+ 810us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:14:06.900739 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:14:06.900743 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:14:06.900923 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:14:06.901522 (+ 599us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:14:06.901530 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:14:06.901537 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:14:06.902131 (+ 594us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:14:06.902421 (+ 290us) server_negotiation.cc:1035] Waiting for connection context
0923 20:14:06.903026 (+ 605us) server_negotiation.cc:299] Negotiation successful
0923 20:14:06.903257 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260923 20:14:06.904307 21587 ts_tablet_manager.cc:1939] T 49f7da30562f426ab0b407212e173fa3 P 4f69f10b722f451cb008413449c8a1a3: Deleting consensus metadata
I20260923 20:14:06.905579 21273 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:14:06.905988 21075 catalog_manager.cc:4841] TS 4f69f10b722f451cb008413449c8a1a3 (127.1.170.129:44809): tablet 49f7da30562f426ab0b407212e173fa3 (table test-table [id=44df16e769d74cafb24bd8eb4271b851]) successfully deleted
I20260923 20:14:07.039094 1706 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqUuku6/build/asan/bin/kudu with pid 21150
I20260923 20:14:07.057180 1706 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskqUuku6/build/asan/bin/kudu pid 21284
Sep 23 20:14:07 dist-test-slave-th4g krb5kdc[21044](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194444, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.170.130@KRBTEST.COM
I20260923 20:14:07.074540 21561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:14:07.058061 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36242 (local address 127.1.170.130:37617)
0923 20:14:07.058232 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0923 20:14:07.058236 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 20:14:07.058254 (+ 18us) server_negotiation.cc:407] Connection header received
0923 20:14:07.058418 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:14:07.058422 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:14:07.058488 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:14:07.058570 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:14:07.060179 (+ 1609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:14:07.061061 (+ 882us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:14:07.062988 (+ 1927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:14:07.063395 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:14:07.063492 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:14:07.068663 (+ 5171us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:14:07.068677 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:14:07.068692 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:14:07.068726 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:14:07.071777 (+ 3051us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:14:07.072604 (+ 827us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:14:07.072611 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:14:07.072613 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:14:07.072745 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:14:07.073128 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:14:07.073131 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:14:07.073133 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:14:07.073481 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:14:07.073690 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0923 20:14:07.074070 (+ 380us) server_negotiation.cc:299] Negotiation successful
0923 20:14:07.074228 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260923 20:14:07.075302 21407 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:14:07.191006 1706 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqUuku6/build/asan/bin/kudu with pid 21284
I20260923 20:14:07.209188 1706 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskqUuku6/build/asan/bin/kudu pid 21418
Sep 23 20:14:07 dist-test-slave-th4g krb5kdc[21044](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194444, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.170.131@KRBTEST.COM
I20260923 20:14:07.227198 21562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:14:07.210002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60592 (local address 127.1.170.131:38231)
0923 20:14:07.210177 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0923 20:14:07.210182 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 20:14:07.210200 (+ 18us) server_negotiation.cc:407] Connection header received
0923 20:14:07.210396 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:14:07.210400 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:14:07.210498 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:14:07.210580 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:14:07.212384 (+ 1804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:14:07.213207 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:14:07.215127 (+ 1920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:14:07.215570 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:14:07.215676 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:14:07.220983 (+ 5307us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:14:07.220996 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:14:07.221012 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:14:07.221055 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:14:07.224292 (+ 3237us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:14:07.225033 (+ 741us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:14:07.225040 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:14:07.225043 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:14:07.225195 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:14:07.225579 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:14:07.225583 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:14:07.225585 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:14:07.226005 (+ 420us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:14:07.226324 (+ 319us) server_negotiation.cc:1035] Waiting for connection context
0923 20:14:07.226608 (+ 284us) server_negotiation.cc:299] Negotiation successful
0923 20:14:07.226829 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260923 20:14:07.227890 21541 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:14:07.352339 1706 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqUuku6/build/asan/bin/kudu with pid 21418
I20260923 20:14:07.372540 1706 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskqUuku6/build/asan/bin/kudu pid 21057
I20260923 20:14:07.373651 21119 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:14:07.485558 1706 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqUuku6/build/asan/bin/kudu with pid 21057
2026-09-23T20:14:07Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (4449 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-09-23T20:14:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T20:14:07Z Disabled control of system clock
I20260923 20:14:07.566782 1706 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqUuku6/build/asan/bin/kudu
/tmp/dist-test-taskqUuku6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790194121577027-1706-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790194121577027-1706-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790194121577027-1706-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790194121577027-1706-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.170.190:36285
--webserver_interface=127.1.170.190
--webserver_port=0
--builtin_ntp_servers=127.1.170.148:35137
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.170.190:36285
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260923 20:14:07.948225 21598 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:14:07.948553 21598 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:14:07.948711 21598 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:14:07.958371 21598 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 20:14:07.958513 21598 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:14:07.958551 21598 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:14:07.958649 21598 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 20:14:07.958732 21598 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 20:14:07.970358 21598 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.170.148:35137
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790194121577027-1706-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790194121577027-1706-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.170.190:36285
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.170.190:36285
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790194121577027-1706-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.170.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790194121577027-1706-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:14:07.971773 21598 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:14:07.973658 21598 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:14:07.986514 21603 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:14:07.986478 21606 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:14:07.986439 21604 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:14:07.987154 21598 server_base.cc:1034] running on GCE node
I20260923 20:14:07.988054 21598 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:14:07.989560 21598 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:14:07.990803 21598 hybrid_clock.cc:648] HybridClock initialized: now 1790194447990705 us; error 92 us; skew 500 ppm
I20260923 20:14:07.991238 21598 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260923 20:14:07.993840 21598 webserver.cc:460] Webserver started at http://127.1.170.190:42927/ using document root <none> and password file <none>
I20260923 20:14:07.994620 21598 fs_manager.cc:359] Metadata directory not provided
I20260923 20:14:07.994710 21598 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:14:07.995040 21598 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:14:07.998082 21598 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790194121577027-1706-0/minicluster-data/master-0/data/instance:
uuid: "83401988a73c41d68a46098bb1023fe2"
format_stamp: "Formatted at 2026-09-23 20:14:07 on dist-test-slave-th4g"
server_key: "ebd59d275a7b59d2be0dcf9f16d6b66e"
server_key_iv: "c629f81f76b590d68c178790e111e927"
server_key_version: "encryptionkey@0"
I20260923 20:14:07.998921 21598 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790194121577027-1706-0/minicluster-data/master-0/wal/instance:
uuid: "83401988a73c41d68a46098bb1023fe2"
format_stamp: "Formatted at 2026-09-23 20:14:07 on dist-test-slave-th4g"
server_key: "ebd59d275a7b59d2be0dcf9f16d6b66e"
server_key_iv: "c629f81f76b590d68c178790e111e927"
server_key_version: "encryptionkey@0"
I20260923 20:14:08.004318 21598 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260923 20:14:08.008047 21612 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:14:08.009599 21598 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260923 20:14:08.009814 21598 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790194121577027-1706-0/minicluster-data/master-0/data,/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790194121577027-1706-0/minicluster-data/master-0/wal
uuid: "83401988a73c41d68a46098bb1023fe2"
format_stamp: "Formatted at 2026-09-23 20:14:07 on dist-test-slave-th4g"
server_key: "ebd59d275a7b59d2be0dcf9f16d6b66e"
server_key_iv: "c629f81f76b590d68c178790e111e927"
server_key_version: "encryptionkey@0"
I20260923 20:14:08.010115 21598 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790194121577027-1706-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790194121577027-1706-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790194121577027-1706-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:14:08.031431 21598 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:14:08.032538 21598 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:14:08.032949 21598 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:14:08.056151 21598 rpc_server.cc:307] RPC server started. Bound to: 127.1.170.190:36285
I20260923 20:14:08.056180 21663 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.170.190:36285 every 8 connection(s)
I20260923 20:14:08.057577 21598 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790194121577027-1706-0/minicluster-data/master-0/data/info.pb
I20260923 20:14:08.061486 21664 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:14:08.065215 1706 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqUuku6/build/asan/bin/kudu as pid 21598
I20260923 20:14:08.065412 1706 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790194121577027-1706-0/minicluster-data/master-0/wal/instance
I20260923 20:14:08.065867 1706 external_mini_cluster.cc:1428] Setting key c1ffb70d705173f89427e5b53cfc9c44
I20260923 20:14:08.074378 21664 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 83401988a73c41d68a46098bb1023fe2: Bootstrap starting.
I20260923 20:14:08.077989 21664 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 83401988a73c41d68a46098bb1023fe2: Neither blocks nor log segments found. Creating new log.
I20260923 20:14:08.079277 21664 log.cc:824] T 00000000000000000000000000000000 P 83401988a73c41d68a46098bb1023fe2: Log is configured to *not* fsync() on all Append() calls
I20260923 20:14:08.079802 21667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:14:08.067856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50030 (local address 127.1.170.190:36285)
0923 20:14:08.068893 (+ 1037us) server_negotiation.cc:206] Beginning negotiation
0923 20:14:08.068906 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0923 20:14:08.068952 (+ 46us) server_negotiation.cc:407] Connection header received
0923 20:14:08.070763 (+ 1811us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:14:08.070785 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:14:08.071321 (+ 536us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:14:08.071731 (+ 410us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:14:08.072374 (+ 643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:14:08.073638 (+ 1264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:14:08.075116 (+ 1478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:14:08.075829 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:14:08.076061 (+ 232us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:14:08.077265 (+ 1204us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:14:08.077273 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:14:08.077289 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 20:14:08.077930 (+ 641us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:14:08.078045 (+ 115us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 20:14:08.078068 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:14:08.078203 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0923 20:14:08.078432 (+ 229us) server_negotiation.cc:299] Negotiation successful
0923 20:14:08.078775 (+ 343us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":592,"thread_start_us":365,"threads_started":1}
I20260923 20:14:08.083240 21664 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 83401988a73c41d68a46098bb1023fe2: No bootstrap required, opened a new log
I20260923 20:14:08.091998 21664 raft_consensus.cc:348] T 00000000000000000000000000000000 P 83401988a73c41d68a46098bb1023fe2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83401988a73c41d68a46098bb1023fe2" member_type: VOTER last_known_addr { host: "127.1.170.190" port: 36285 } }
I20260923 20:14:08.092448 21664 raft_consensus.cc:374] T 00000000000000000000000000000000 P 83401988a73c41d68a46098bb1023fe2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:14:08.092559 21664 raft_consensus.cc:729] T 00000000000000000000000000000000 P 83401988a73c41d68a46098bb1023fe2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83401988a73c41d68a46098bb1023fe2, State: Initialized, Role: FOLLOWER
I20260923 20:14:08.093225 21664 consensus_queue.cc:260] T 00000000000000000000000000000000 P 83401988a73c41d68a46098bb1023fe2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83401988a73c41d68a46098bb1023fe2" member_type: VOTER last_known_addr { host: "127.1.170.190" port: 36285 } }
I20260923 20:14:08.093480 21664 raft_consensus.cc:388] T 00000000000000000000000000000000 P 83401988a73c41d68a46098bb1023fe2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 20:14:08.093667 21664 raft_consensus.cc:482] T 00000000000000000000000000000000 P 83401988a73c41d68a46098bb1023fe2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 20:14:08.093909 21664 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 83401988a73c41d68a46098bb1023fe2 [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:14:08.096908 21664 raft_consensus.cc:504] T 00000000000000000000000000000000 P 83401988a73c41d68a46098bb1023fe2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83401988a73c41d68a46098bb1023fe2" member_type: VOTER last_known_addr { host: "127.1.170.190" port: 36285 } }
I20260923 20:14:08.097770 21664 leader_election.cc:302] T 00000000000000000000000000000000 P 83401988a73c41d68a46098bb1023fe2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 83401988a73c41d68a46098bb1023fe2; no voters:
I20260923 20:14:08.098958 21664 leader_election.cc:288] T 00000000000000000000000000000000 P 83401988a73c41d68a46098bb1023fe2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 20:14:08.099061 21669 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 83401988a73c41d68a46098bb1023fe2 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 20:14:08.099928 21669 raft_consensus.cc:686] T 00000000000000000000000000000000 P 83401988a73c41d68a46098bb1023fe2 [term 1 LEADER]: Becoming Leader. State: Replica: 83401988a73c41d68a46098bb1023fe2, State: Running, Role: LEADER
I20260923 20:14:08.100492 21669 consensus_queue.cc:237] T 00000000000000000000000000000000 P 83401988a73c41d68a46098bb1023fe2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83401988a73c41d68a46098bb1023fe2" member_type: VOTER last_known_addr { host: "127.1.170.190" port: 36285 } }
I20260923 20:14:08.101392 21664 sys_catalog.cc:564] T 00000000000000000000000000000000 P 83401988a73c41d68a46098bb1023fe2 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 20:14:08.103384 21670 sys_catalog.cc:455] T 00000000000000000000000000000000 P 83401988a73c41d68a46098bb1023fe2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "83401988a73c41d68a46098bb1023fe2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83401988a73c41d68a46098bb1023fe2" member_type: VOTER last_known_addr { host: "127.1.170.190" port: 36285 } } }
I20260923 20:14:08.103796 21670 sys_catalog.cc:458] T 00000000000000000000000000000000 P 83401988a73c41d68a46098bb1023fe2 [sys.catalog]: This master's current role is: LEADER
I20260923 20:14:08.104230 21671 sys_catalog.cc:455] T 00000000000000000000000000000000 P 83401988a73c41d68a46098bb1023fe2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 83401988a73c41d68a46098bb1023fe2. Latest consensus state: current_term: 1 leader_uuid: "83401988a73c41d68a46098bb1023fe2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83401988a73c41d68a46098bb1023fe2" member_type: VOTER last_known_addr { host: "127.1.170.190" port: 36285 } } }
I20260923 20:14:08.104506 21671 sys_catalog.cc:458] T 00000000000000000000000000000000 P 83401988a73c41d68a46098bb1023fe2 [sys.catalog]: This master's current role is: LEADER
I20260923 20:14:08.110394 21675 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 20:14:08.116451 21675 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 20:14:08.129504 21675 catalog_manager.cc:1347] Generated new cluster ID: d55af1e4572e4531a9053ea5490668c3
I20260923 20:14:08.129680 21675 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 20:14:08.158355 21675 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 20:14:08.160285 21675 catalog_manager.cc:1504] Loading token signing keys...
I20260923 20:14:08.172094 21675 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 83401988a73c41d68a46098bb1023fe2: Generated new TSK 0
I20260923 20:14:08.173659 21675 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 20:14:08.190713 1706 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqUuku6/build/asan/bin/kudu
/tmp/dist-test-taskqUuku6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790194121577027-1706-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790194121577027-1706-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790194121577027-1706-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790194121577027-1706-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.170.129:0
--local_ip_for_outbound_sockets=127.1.170.129
--webserver_interface=127.1.170.129
--webserver_port=0
--tserver_master_addrs=127.1.170.190:36285
--builtin_ntp_servers=127.1.170.148:35137
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260923 20:14:08.547709 21688 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:14:08.548058 21688 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:14:08.548223 21688 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:14:08.556123 21688 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:14:08.556282 21688 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:14:08.556469 21688 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.170.129
I20260923 20:14:08.567826 21688 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.170.148:35137
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790194121577027-1706-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790194121577027-1706-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.170.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790194121577027-1706-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.170.129
--webserver_port=0
--tserver_master_addrs=127.1.170.190:36285
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.170.129
--log_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790194121577027-1706-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:14:08.569186 21688 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:14:08.571094 21688 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:14:08.585222 21696 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:14:08.585217 21694 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:14:08.585513 21693 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:14:08.585714 21688 server_base.cc:1034] running on GCE node
I20260923 20:14:08.586647 21688 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:14:08.587670 21688 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:14:08.588964 21688 hybrid_clock.cc:648] HybridClock initialized: now 1790194448588915 us; error 29 us; skew 500 ppm
I20260923 20:14:08.589285 21688 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260923 20:14:08.591946 21688 webserver.cc:460] Webserver started at http://127.1.170.129:40765/ using document root <none> and password file <none>
I20260923 20:14:08.592656 21688 fs_manager.cc:359] Metadata directory not provided
I20260923 20:14:08.592765 21688 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:14:08.593043 21688 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:14:08.596139 21688 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790194121577027-1706-0/minicluster-data/ts-0/data/instance:
uuid: "93f65129f65f48dcb4f740d3d0cb2d93"
format_stamp: "Formatted at 2026-09-23 20:14:08 on dist-test-slave-th4g"
server_key: "2c6d7fd59a46882926f477efc0576455"
server_key_iv: "17e8b4e1eeb1eeb57e53a80a7111fb89"
server_key_version: "encryptionkey@0"
I20260923 20:14:08.596920 21688 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790194121577027-1706-0/minicluster-data/ts-0/wal/instance:
uuid: "93f65129f65f48dcb4f740d3d0cb2d93"
format_stamp: "Formatted at 2026-09-23 20:14:08 on dist-test-slave-th4g"
server_key: "2c6d7fd59a46882926f477efc0576455"
server_key_iv: "17e8b4e1eeb1eeb57e53a80a7111fb89"
server_key_version: "encryptionkey@0"
I20260923 20:14:08.602486 21688 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260923 20:14:08.606403 21702 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:14:08.608479 21688 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260923 20:14:08.608678 21688 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790194121577027-1706-0/minicluster-data/ts-0/data,/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790194121577027-1706-0/minicluster-data/ts-0/wal
uuid: "93f65129f65f48dcb4f740d3d0cb2d93"
format_stamp: "Formatted at 2026-09-23 20:14:08 on dist-test-slave-th4g"
server_key: "2c6d7fd59a46882926f477efc0576455"
server_key_iv: "17e8b4e1eeb1eeb57e53a80a7111fb89"
server_key_version: "encryptionkey@0"
I20260923 20:14:08.608906 21688 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790194121577027-1706-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790194121577027-1706-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790194121577027-1706-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:14:08.635154 21688 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:14:08.636049 21688 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:14:08.636369 21688 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:14:08.637395 21688 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 20:14:08.638927 21688 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 20:14:08.639019 21688 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 20:14:08.639166 21688 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 20:14:08.639246 21688 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 20:14:08.677553 21688 rpc_server.cc:307] RPC server started. Bound to: 127.1.170.129:42153
I20260923 20:14:08.677729 21814 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.170.129:42153 every 8 connection(s)
I20260923 20:14:08.678987 21688 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790194121577027-1706-0/minicluster-data/ts-0/data/info.pb
I20260923 20:14:08.682338 1706 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqUuku6/build/asan/bin/kudu as pid 21688
I20260923 20:14:08.682511 1706 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790194121577027-1706-0/minicluster-data/ts-0/wal/instance
I20260923 20:14:08.683103 1706 external_mini_cluster.cc:1428] Setting key 064755ffb06ca2030cde5dc5ea7d4e7f
I20260923 20:14:08.686021 1706 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqUuku6/build/asan/bin/kudu
/tmp/dist-test-taskqUuku6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790194121577027-1706-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790194121577027-1706-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790194121577027-1706-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790194121577027-1706-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.170.130:0
--local_ip_for_outbound_sockets=127.1.170.130
--webserver_interface=127.1.170.130
--webserver_port=0
--tserver_master_addrs=127.1.170.190:36285
--builtin_ntp_servers=127.1.170.148:35137
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260923 20:14:08.694181 21817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:14:08.681992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.170.129:33253 (local address 127.1.170.190:36285)
0923 20:14:08.682657 (+ 665us) server_negotiation.cc:206] Beginning negotiation
0923 20:14:08.682666 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 20:14:08.683850 (+ 1184us) server_negotiation.cc:407] Connection header received
0923 20:14:08.685910 (+ 2060us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:14:08.685921 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:14:08.686014 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:14:08.686143 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:14:08.687399 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:14:08.688618 (+ 1219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:14:08.690487 (+ 1869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:14:08.691252 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:14:08.691409 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:14:08.692945 (+ 1536us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:14:08.692960 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:14:08.692968 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 20:14:08.692997 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:14:08.693048 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 20:14:08.693066 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:14:08.693166 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0923 20:14:08.693665 (+ 499us) server_negotiation.cc:299] Negotiation successful
0923 20:14:08.693788 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":495,"thread_start_us":345,"threads_started":1}
I20260923 20:14:08.696127 21818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:14:08.682518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.190:36285 (local address 127.1.170.129:33253)
0923 20:14:08.683650 (+ 1132us) negotiation.cc:107] Waiting for socket to connect
0923 20:14:08.683711 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0923 20:14:08.685575 (+ 1864us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:14:08.686466 (+ 891us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:14:08.686475 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:14:08.686578 (+ 103us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:14:08.687203 (+ 625us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:14:08.687221 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:14:08.688802 (+ 1581us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:14:08.688806 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:14:08.690328 (+ 1522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:14:08.690342 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:14:08.691509 (+ 1167us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:14:08.691515 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:14:08.691739 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:14:08.692568 (+ 829us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 20:14:08.692571 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:14:08.692615 (+ 44us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 20:14:08.692721 (+ 106us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:14:08.693295 (+ 574us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:14:08.693315 (+ 20us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:14:08.693327 (+ 12us) client_negotiation.cc:769] Sending connection context
0923 20:14:08.693622 (+ 295us) client_negotiation.cc:240] Negotiation successful
0923 20:14:08.694618 (+ 996us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":580,"thread_start_us":359,"threads_started":1}
I20260923 20:14:08.698257 21815 heartbeater.cc:347] Connected to a master server at 127.1.170.190:36285
I20260923 20:14:08.698634 21815 heartbeater.cc:464] Registering TS with master...
I20260923 20:14:08.699605 21815 heartbeater.cc:511] Master 127.1.170.190:36285 requested a full tablet report, sending...
I20260923 20:14:08.702003 21629 ts_manager.cc:194] Registered new tserver with Master: 93f65129f65f48dcb4f740d3d0cb2d93 (127.1.170.129:42153)
I20260923 20:14:08.703584 21629 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.170.129:33253
W20260923 20:14:09.049233 21819 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:14:09.049566 21819 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:14:09.049741 21819 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:14:09.060168 21819 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:14:09.060305 21819 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:14:09.060472 21819 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.170.130
I20260923 20:14:09.075502 21819 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.170.148:35137
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790194121577027-1706-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790194121577027-1706-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.170.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790194121577027-1706-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.170.130
--webserver_port=0
--tserver_master_addrs=127.1.170.190:36285
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.170.130
--log_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790194121577027-1706-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:14:09.077127 21819 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:14:09.079406 21819 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:14:09.094385 21825 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:14:09.094498 21827 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:14:09.094475 21824 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:14:09.095089 21819 server_base.cc:1034] running on GCE node
I20260923 20:14:09.095867 21819 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:14:09.097114 21819 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:14:09.098489 21819 hybrid_clock.cc:648] HybridClock initialized: now 1790194449098373 us; error 74 us; skew 500 ppm
I20260923 20:14:09.098982 21819 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260923 20:14:09.102676 21819 webserver.cc:460] Webserver started at http://127.1.170.130:39807/ using document root <none> and password file <none>
I20260923 20:14:09.103511 21819 fs_manager.cc:359] Metadata directory not provided
I20260923 20:14:09.103614 21819 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:14:09.103909 21819 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:14:09.107209 21819 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790194121577027-1706-0/minicluster-data/ts-1/data/instance:
uuid: "e757ee402fd040fe9a5113b623aad3f2"
format_stamp: "Formatted at 2026-09-23 20:14:09 on dist-test-slave-th4g"
server_key: "4bca35de3bd40a52f1b4253b95f4c189"
server_key_iv: "43bb342da435bfddc9ddc6a780b588e2"
server_key_version: "encryptionkey@0"
I20260923 20:14:09.108062 21819 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790194121577027-1706-0/minicluster-data/ts-1/wal/instance:
uuid: "e757ee402fd040fe9a5113b623aad3f2"
format_stamp: "Formatted at 2026-09-23 20:14:09 on dist-test-slave-th4g"
server_key: "4bca35de3bd40a52f1b4253b95f4c189"
server_key_iv: "43bb342da435bfddc9ddc6a780b588e2"
server_key_version: "encryptionkey@0"
I20260923 20:14:09.114045 21819 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260923 20:14:09.120278 21833 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:14:09.122510 21819 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.006s sys 0.000s
I20260923 20:14:09.122740 21819 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790194121577027-1706-0/minicluster-data/ts-1/data,/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790194121577027-1706-0/minicluster-data/ts-1/wal
uuid: "e757ee402fd040fe9a5113b623aad3f2"
format_stamp: "Formatted at 2026-09-23 20:14:09 on dist-test-slave-th4g"
server_key: "4bca35de3bd40a52f1b4253b95f4c189"
server_key_iv: "43bb342da435bfddc9ddc6a780b588e2"
server_key_version: "encryptionkey@0"
I20260923 20:14:09.123075 21819 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790194121577027-1706-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790194121577027-1706-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790194121577027-1706-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:14:09.138666 21819 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:14:09.139539 21819 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:14:09.139855 21819 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:14:09.140977 21819 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 20:14:09.142722 21819 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 20:14:09.142836 21819 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 20:14:09.142973 21819 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 20:14:09.143054 21819 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 20:14:09.178733 21819 rpc_server.cc:307] RPC server started. Bound to: 127.1.170.130:43839
I20260923 20:14:09.178771 21945 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.170.130:43839 every 8 connection(s)
I20260923 20:14:09.180223 21819 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790194121577027-1706-0/minicluster-data/ts-1/data/info.pb
I20260923 20:14:09.190119 1706 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqUuku6/build/asan/bin/kudu as pid 21819
I20260923 20:14:09.190325 1706 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790194121577027-1706-0/minicluster-data/ts-1/wal/instance
I20260923 20:14:09.190793 1706 external_mini_cluster.cc:1428] Setting key 61e01ff411fe2078db9e0f11bfdeeba3
I20260923 20:14:09.194571 1706 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqUuku6/build/asan/bin/kudu
/tmp/dist-test-taskqUuku6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790194121577027-1706-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790194121577027-1706-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790194121577027-1706-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790194121577027-1706-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.170.131:0
--local_ip_for_outbound_sockets=127.1.170.131
--webserver_interface=127.1.170.131
--webserver_port=0
--tserver_master_addrs=127.1.170.190:36285
--builtin_ntp_servers=127.1.170.148:35137
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260923 20:14:09.195516 21817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:14:09.183183 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.170.130:58495 (local address 127.1.170.190:36285)
0923 20:14:09.183387 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0923 20:14:09.183392 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 20:14:09.184902 (+ 1510us) server_negotiation.cc:407] Connection header received
0923 20:14:09.186871 (+ 1969us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:14:09.186877 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:14:09.186971 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:14:09.187084 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:14:09.188191 (+ 1107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:14:09.188934 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:14:09.191511 (+ 2577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:14:09.192128 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:14:09.192472 (+ 344us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:14:09.194333 (+ 1861us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:14:09.194339 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:14:09.194343 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 20:14:09.194368 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:14:09.194419 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 20:14:09.194434 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:14:09.194534 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0923 20:14:09.194895 (+ 361us) server_negotiation.cc:299] Negotiation successful
0923 20:14:09.195031 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":50,"server-negotiator.queue_time_us":83}
I20260923 20:14:09.197806 21948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:14:09.183730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.190:36285 (local address 127.1.170.130:58495)
0923 20:14:09.184725 (+ 995us) negotiation.cc:107] Waiting for socket to connect
0923 20:14:09.184768 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0923 20:14:09.186570 (+ 1802us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:14:09.187323 (+ 753us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:14:09.187338 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:14:09.187430 (+ 92us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:14:09.188012 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:14:09.188029 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:14:09.189106 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:14:09.189110 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:14:09.191238 (+ 2128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:14:09.191268 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:14:09.192526 (+ 1258us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:14:09.192532 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:14:09.192805 (+ 273us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:14:09.193848 (+ 1043us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 20:14:09.193852 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:14:09.193933 (+ 81us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 20:14:09.194049 (+ 116us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:14:09.194574 (+ 525us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:14:09.194588 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:14:09.194599 (+ 11us) client_negotiation.cc:769] Sending connection context
0923 20:14:09.195743 (+ 1144us) client_negotiation.cc:240] Negotiation successful
0923 20:14:09.196262 (+ 519us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":513,"thread_start_us":321,"threads_started":1}
I20260923 20:14:09.199723 21946 heartbeater.cc:347] Connected to a master server at 127.1.170.190:36285
I20260923 20:14:09.200086 21946 heartbeater.cc:464] Registering TS with master...
I20260923 20:14:09.201018 21946 heartbeater.cc:511] Master 127.1.170.190:36285 requested a full tablet report, sending...
I20260923 20:14:09.202883 21629 ts_manager.cc:194] Registered new tserver with Master: e757ee402fd040fe9a5113b623aad3f2 (127.1.170.130:43839)
I20260923 20:14:09.204118 21629 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.170.130:58495
W20260923 20:14:09.548164 21949 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:14:09.548528 21949 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:14:09.548704 21949 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:14:09.556699 21949 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:14:09.556859 21949 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:14:09.557039 21949 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.170.131
I20260923 20:14:09.566607 21949 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.170.148:35137
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790194121577027-1706-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790194121577027-1706-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.170.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790194121577027-1706-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.170.131
--webserver_port=0
--tserver_master_addrs=127.1.170.190:36285
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.170.131
--log_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790194121577027-1706-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:14:09.567970 21949 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:14:09.570101 21949 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:14:09.584501 21954 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:14:09.584627 21955 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:14:09.584662 21957 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:14:09.585464 21949 server_base.cc:1034] running on GCE node
I20260923 20:14:09.586172 21949 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:14:09.587190 21949 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:14:09.588440 21949 hybrid_clock.cc:648] HybridClock initialized: now 1790194449588393 us; error 41 us; skew 500 ppm
I20260923 20:14:09.588795 21949 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260923 20:14:09.591588 21949 webserver.cc:460] Webserver started at http://127.1.170.131:40915/ using document root <none> and password file <none>
I20260923 20:14:09.592342 21949 fs_manager.cc:359] Metadata directory not provided
I20260923 20:14:09.592433 21949 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:14:09.592842 21949 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:14:09.595899 21949 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790194121577027-1706-0/minicluster-data/ts-2/data/instance:
uuid: "4acafeb234cc492e9fcfec9dbf23d23f"
format_stamp: "Formatted at 2026-09-23 20:14:09 on dist-test-slave-th4g"
server_key: "1c5103319b402477c483cbbb9b4793cb"
server_key_iv: "e6690e52661fc0d1ad2b42c0d75f5300"
server_key_version: "encryptionkey@0"
I20260923 20:14:09.596707 21949 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790194121577027-1706-0/minicluster-data/ts-2/wal/instance:
uuid: "4acafeb234cc492e9fcfec9dbf23d23f"
format_stamp: "Formatted at 2026-09-23 20:14:09 on dist-test-slave-th4g"
server_key: "1c5103319b402477c483cbbb9b4793cb"
server_key_iv: "e6690e52661fc0d1ad2b42c0d75f5300"
server_key_version: "encryptionkey@0"
I20260923 20:14:09.602268 21949 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260923 20:14:09.606141 21963 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:14:09.607554 21949 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260923 20:14:09.607738 21949 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790194121577027-1706-0/minicluster-data/ts-2/data,/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790194121577027-1706-0/minicluster-data/ts-2/wal
uuid: "4acafeb234cc492e9fcfec9dbf23d23f"
format_stamp: "Formatted at 2026-09-23 20:14:09 on dist-test-slave-th4g"
server_key: "1c5103319b402477c483cbbb9b4793cb"
server_key_iv: "e6690e52661fc0d1ad2b42c0d75f5300"
server_key_version: "encryptionkey@0"
I20260923 20:14:09.607987 21949 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790194121577027-1706-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790194121577027-1706-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790194121577027-1706-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:14:09.629108 21949 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:14:09.630070 21949 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:14:09.630393 21949 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:14:09.631394 21949 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 20:14:09.632934 21949 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 20:14:09.633038 21949 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 20:14:09.633188 21949 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 20:14:09.633276 21949 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 20:14:09.670775 21949 rpc_server.cc:307] RPC server started. Bound to: 127.1.170.131:38677
I20260923 20:14:09.670818 22075 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.170.131:38677 every 8 connection(s)
I20260923 20:14:09.672178 21949 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790194121577027-1706-0/minicluster-data/ts-2/data/info.pb
I20260923 20:14:09.675932 1706 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqUuku6/build/asan/bin/kudu as pid 21949
I20260923 20:14:09.676111 1706 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790194121577027-1706-0/minicluster-data/ts-2/wal/instance
I20260923 20:14:09.676640 1706 external_mini_cluster.cc:1428] Setting key 367b291bb16a0e5deea9e191b16db9e1
I20260923 20:14:09.686151 21817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:14:09.674928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.170.131:39507 (local address 127.1.170.190:36285)
0923 20:14:09.675121 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0923 20:14:09.675125 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 20:14:09.676671 (+ 1546us) server_negotiation.cc:407] Connection header received
0923 20:14:09.679010 (+ 2339us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:14:09.679016 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:14:09.679101 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:14:09.679215 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:14:09.680412 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:14:09.681583 (+ 1171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:14:09.683352 (+ 1769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:14:09.683750 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:14:09.683846 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:14:09.685196 (+ 1350us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:14:09.685202 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:14:09.685206 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 20:14:09.685226 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:14:09.685262 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 20:14:09.685275 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:14:09.685358 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0923 20:14:09.685678 (+ 320us) server_negotiation.cc:299] Negotiation successful
0923 20:14:09.685767 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260923 20:14:09.687111 22078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:14:09.675377 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.190:36285 (local address 127.1.170.131:39507)
0923 20:14:09.676502 (+ 1125us) negotiation.cc:107] Waiting for socket to connect
0923 20:14:09.676546 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0923 20:14:09.678703 (+ 2157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:14:09.679480 (+ 777us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:14:09.679493 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:14:09.679615 (+ 122us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:14:09.680231 (+ 616us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:14:09.680257 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:14:09.681751 (+ 1494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:14:09.681756 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:14:09.683193 (+ 1437us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:14:09.683207 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:14:09.683953 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:14:09.683958 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:14:09.684161 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:14:09.684863 (+ 702us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 20:14:09.684865 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:14:09.684904 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 20:14:09.684989 (+ 85us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:14:09.685407 (+ 418us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:14:09.685416 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:14:09.685423 (+ 7us) client_negotiation.cc:769] Sending connection context
0923 20:14:09.685614 (+ 191us) client_negotiation.cc:240] Negotiation successful
0923 20:14:09.686038 (+ 424us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":553,"thread_start_us":339,"threads_started":1}
I20260923 20:14:09.688607 22076 heartbeater.cc:347] Connected to a master server at 127.1.170.190:36285
I20260923 20:14:09.688944 22076 heartbeater.cc:464] Registering TS with master...
I20260923 20:14:09.689716 22076 heartbeater.cc:511] Master 127.1.170.190:36285 requested a full tablet report, sending...
I20260923 20:14:09.691214 21629 ts_manager.cc:194] Registered new tserver with Master: 4acafeb234cc492e9fcfec9dbf23d23f (127.1.170.131:38677)
I20260923 20:14:09.692263 21629 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.170.131:39507
I20260923 20:14:09.692371 1706 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260923 20:14:09.703850 21817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:14:09.696765 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50042 (local address 127.1.170.190:36285)
0923 20:14:09.696968 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0923 20:14:09.696973 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 20:14:09.697499 (+ 526us) server_negotiation.cc:407] Connection header received
0923 20:14:09.697846 (+ 347us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:14:09.697855 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:14:09.697986 (+ 131us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:14:09.698118 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:14:09.698680 (+ 562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:14:09.699404 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:14:09.701179 (+ 1775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:14:09.701860 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:14:09.702049 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:14:09.703136 (+ 1087us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:14:09.703141 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:14:09.703144 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 20:14:09.703164 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:14:09.703195 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 20:14:09.703206 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:14:09.703285 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0923 20:14:09.703533 (+ 248us) server_negotiation.cc:299] Negotiation successful
0923 20:14:09.703612 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260923 20:14:09.707582 21815 heartbeater.cc:503] Master 127.1.170.190:36285 was elected leader, sending a full tablet report...
I20260923 20:14:09.708937 21629 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50042:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 20:14:09.717312 21629 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 20:14:09.741009 22088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:14:09.730414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.131:38677 (local address 127.0.0.1:35840)
0923 20:14:09.731524 (+ 1110us) negotiation.cc:107] Waiting for socket to connect
0923 20:14:09.731570 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0923 20:14:09.731778 (+ 208us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:14:09.732745 (+ 967us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:14:09.732752 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:14:09.732788 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:14:09.733362 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:14:09.733381 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:14:09.735378 (+ 1997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:14:09.735385 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:14:09.737727 (+ 2342us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:14:09.737741 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:14:09.740059 (+ 2318us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:14:09.740067 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:14:09.740262 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:14:09.740326 (+ 64us) client_negotiation.cc:769] Sending connection context
0923 20:14:09.740453 (+ 127us) client_negotiation.cc:240] Negotiation successful
0923 20:14:09.740651 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":826,"thread_start_us":484,"threads_started":1}
I20260923 20:14:09.742112 22089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:14:09.730601 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35840 (local address 127.1.170.131:38677)
0923 20:14:09.732045 (+ 1444us) server_negotiation.cc:206] Beginning negotiation
0923 20:14:09.732052 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:14:09.732103 (+ 51us) server_negotiation.cc:407] Connection header received
0923 20:14:09.732208 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:14:09.732223 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:14:09.732497 (+ 274us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:14:09.732745 (+ 248us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:14:09.733575 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:14:09.735218 (+ 1643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:14:09.737973 (+ 2755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:14:09.739498 (+ 1525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:14:09.739638 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:14:09.739925 (+ 287us) server_negotiation.cc:1035] Waiting for connection context
0923 20:14:09.741553 (+ 1628us) server_negotiation.cc:299] Negotiation successful
0923 20:14:09.741735 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1250,"thread_start_us":1095,"threads_started":1}
I20260923 20:14:09.743062 22090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:14:09.731947 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.129:42153 (local address 127.0.0.1:44562)
0923 20:14:09.732426 (+ 479us) negotiation.cc:107] Waiting for socket to connect
0923 20:14:09.732443 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0923 20:14:09.732555 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:14:09.734555 (+ 2000us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:14:09.734559 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:14:09.734582 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:14:09.734961 (+ 379us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:14:09.734971 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:14:09.737923 (+ 2952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:14:09.737929 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:14:09.740022 (+ 2093us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:14:09.740039 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:14:09.742268 (+ 2229us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:14:09.742274 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:14:09.742492 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:14:09.742516 (+ 24us) client_negotiation.cc:769] Sending connection context
0923 20:14:09.742631 (+ 115us) client_negotiation.cc:240] Negotiation successful
0923 20:14:09.742756 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":365,"thread_start_us":778,"threads_started":1}
I20260923 20:14:09.743201 22091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:14:09.732535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44562 (local address 127.1.170.129:42153)
0923 20:14:09.733767 (+ 1232us) server_negotiation.cc:206] Beginning negotiation
0923 20:14:09.733777 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0923 20:14:09.733806 (+ 29us) server_negotiation.cc:407] Connection header received
0923 20:14:09.733927 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:14:09.733934 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:14:09.734164 (+ 230us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:14:09.734337 (+ 173us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:14:09.735147 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:14:09.737734 (+ 2587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:14:09.740241 (+ 2507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:14:09.741722 (+ 1481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:14:09.741847 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:14:09.742140 (+ 293us) server_negotiation.cc:1035] Waiting for connection context
0923 20:14:09.742676 (+ 536us) server_negotiation.cc:299] Negotiation successful
0923 20:14:09.742833 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1051,"thread_start_us":899,"threads_started":1}
I20260923 20:14:09.747629 22011 tablet_service.cc:1467] Processing CreateTablet for tablet 82056ba9b19543dfa61cb8c444a8eae1 (DEFAULT_TABLE table=test-table [id=edc1ae7cb17c40889f6492471d47a112]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 20:14:09.747637 21750 tablet_service.cc:1467] Processing CreateTablet for tablet 82056ba9b19543dfa61cb8c444a8eae1 (DEFAULT_TABLE table=test-table [id=edc1ae7cb17c40889f6492471d47a112]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 20:14:09.749033 21750 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 82056ba9b19543dfa61cb8c444a8eae1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:14:09.749064 22011 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 82056ba9b19543dfa61cb8c444a8eae1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:14:09.751591 22088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:14:09.733029 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.130:43839 (local address 127.0.0.1:41398)
0923 20:14:09.741267 (+ 8238us) negotiation.cc:107] Waiting for socket to connect
0923 20:14:09.741287 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0923 20:14:09.741385 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:14:09.744420 (+ 3035us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:14:09.744424 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:14:09.744453 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:14:09.744624 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:14:09.744639 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:14:09.746711 (+ 2072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:14:09.746717 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:14:09.748528 (+ 1811us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:14:09.748546 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:14:09.750739 (+ 2193us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:14:09.750745 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:14:09.750984 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:14:09.751025 (+ 41us) client_negotiation.cc:769] Sending connection context
0923 20:14:09.751132 (+ 107us) client_negotiation.cc:240] Negotiation successful
0923 20:14:09.751279 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8129,"thread_start_us":10494,"threads_started":1}
I20260923 20:14:09.751921 22093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:14:09.733549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41398 (local address 127.1.170.130:43839)
0923 20:14:09.743799 (+ 10250us) server_negotiation.cc:206] Beginning negotiation
0923 20:14:09.743806 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:14:09.743832 (+ 26us) server_negotiation.cc:407] Connection header received
0923 20:14:09.743917 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:14:09.743924 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:14:09.744223 (+ 299us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:14:09.744438 (+ 215us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:14:09.744793 (+ 355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:14:09.746541 (+ 1748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:14:09.748729 (+ 2188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:14:09.750265 (+ 1536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:14:09.750402 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:14:09.750634 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0923 20:14:09.751202 (+ 568us) server_negotiation.cc:299] Negotiation successful
0923 20:14:09.751374 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10117,"thread_start_us":10037,"threads_started":1}
I20260923 20:14:09.755842 21881 tablet_service.cc:1467] Processing CreateTablet for tablet 82056ba9b19543dfa61cb8c444a8eae1 (DEFAULT_TABLE table=test-table [id=edc1ae7cb17c40889f6492471d47a112]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 20:14:09.757388 21881 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 82056ba9b19543dfa61cb8c444a8eae1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:14:09.763532 22094 tablet_bootstrap.cc:492] T 82056ba9b19543dfa61cb8c444a8eae1 P 4acafeb234cc492e9fcfec9dbf23d23f: Bootstrap starting.
I20260923 20:14:09.765228 22095 tablet_bootstrap.cc:492] T 82056ba9b19543dfa61cb8c444a8eae1 P 93f65129f65f48dcb4f740d3d0cb2d93: Bootstrap starting.
I20260923 20:14:09.767392 22094 tablet_bootstrap.cc:654] T 82056ba9b19543dfa61cb8c444a8eae1 P 4acafeb234cc492e9fcfec9dbf23d23f: Neither blocks nor log segments found. Creating new log.
I20260923 20:14:09.768607 22094 log.cc:824] T 82056ba9b19543dfa61cb8c444a8eae1 P 4acafeb234cc492e9fcfec9dbf23d23f: Log is configured to *not* fsync() on all Append() calls
I20260923 20:14:09.769029 22095 tablet_bootstrap.cc:654] T 82056ba9b19543dfa61cb8c444a8eae1 P 93f65129f65f48dcb4f740d3d0cb2d93: Neither blocks nor log segments found. Creating new log.
I20260923 20:14:09.770267 22095 log.cc:824] T 82056ba9b19543dfa61cb8c444a8eae1 P 93f65129f65f48dcb4f740d3d0cb2d93: Log is configured to *not* fsync() on all Append() calls
I20260923 20:14:09.771765 22094 tablet_bootstrap.cc:492] T 82056ba9b19543dfa61cb8c444a8eae1 P 4acafeb234cc492e9fcfec9dbf23d23f: No bootstrap required, opened a new log
I20260923 20:14:09.772024 22094 ts_tablet_manager.cc:1397] T 82056ba9b19543dfa61cb8c444a8eae1 P 4acafeb234cc492e9fcfec9dbf23d23f: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260923 20:14:09.773985 22095 tablet_bootstrap.cc:492] T 82056ba9b19543dfa61cb8c444a8eae1 P 93f65129f65f48dcb4f740d3d0cb2d93: No bootstrap required, opened a new log
I20260923 20:14:09.774533 22095 ts_tablet_manager.cc:1397] T 82056ba9b19543dfa61cb8c444a8eae1 P 93f65129f65f48dcb4f740d3d0cb2d93: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.006s
I20260923 20:14:09.775511 22098 tablet_bootstrap.cc:492] T 82056ba9b19543dfa61cb8c444a8eae1 P e757ee402fd040fe9a5113b623aad3f2: Bootstrap starting.
I20260923 20:14:09.780931 22098 tablet_bootstrap.cc:654] T 82056ba9b19543dfa61cb8c444a8eae1 P e757ee402fd040fe9a5113b623aad3f2: Neither blocks nor log segments found. Creating new log.
I20260923 20:14:09.782317 22098 log.cc:824] T 82056ba9b19543dfa61cb8c444a8eae1 P e757ee402fd040fe9a5113b623aad3f2: Log is configured to *not* fsync() on all Append() calls
I20260923 20:14:09.783074 22094 raft_consensus.cc:348] T 82056ba9b19543dfa61cb8c444a8eae1 P 4acafeb234cc492e9fcfec9dbf23d23f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4acafeb234cc492e9fcfec9dbf23d23f" member_type: VOTER last_known_addr { host: "127.1.170.131" port: 38677 } } peers { permanent_uuid: "93f65129f65f48dcb4f740d3d0cb2d93" member_type: VOTER last_known_addr { host: "127.1.170.129" port: 42153 } } peers { permanent_uuid: "e757ee402fd040fe9a5113b623aad3f2" member_type: VOTER last_known_addr { host: "127.1.170.130" port: 43839 } }
I20260923 20:14:09.783062 22095 raft_consensus.cc:348] T 82056ba9b19543dfa61cb8c444a8eae1 P 93f65129f65f48dcb4f740d3d0cb2d93 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4acafeb234cc492e9fcfec9dbf23d23f" member_type: VOTER last_known_addr { host: "127.1.170.131" port: 38677 } } peers { permanent_uuid: "93f65129f65f48dcb4f740d3d0cb2d93" member_type: VOTER last_known_addr { host: "127.1.170.129" port: 42153 } } peers { permanent_uuid: "e757ee402fd040fe9a5113b623aad3f2" member_type: VOTER last_known_addr { host: "127.1.170.130" port: 43839 } }
I20260923 20:14:09.783480 22094 raft_consensus.cc:374] T 82056ba9b19543dfa61cb8c444a8eae1 P 4acafeb234cc492e9fcfec9dbf23d23f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:14:09.783525 22095 raft_consensus.cc:374] T 82056ba9b19543dfa61cb8c444a8eae1 P 93f65129f65f48dcb4f740d3d0cb2d93 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:14:09.783571 22094 raft_consensus.cc:729] T 82056ba9b19543dfa61cb8c444a8eae1 P 4acafeb234cc492e9fcfec9dbf23d23f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4acafeb234cc492e9fcfec9dbf23d23f, State: Initialized, Role: FOLLOWER
I20260923 20:14:09.783623 22095 raft_consensus.cc:729] T 82056ba9b19543dfa61cb8c444a8eae1 P 93f65129f65f48dcb4f740d3d0cb2d93 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93f65129f65f48dcb4f740d3d0cb2d93, State: Initialized, Role: FOLLOWER
I20260923 20:14:09.784279 22095 consensus_queue.cc:260] T 82056ba9b19543dfa61cb8c444a8eae1 P 93f65129f65f48dcb4f740d3d0cb2d93 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4acafeb234cc492e9fcfec9dbf23d23f" member_type: VOTER last_known_addr { host: "127.1.170.131" port: 38677 } } peers { permanent_uuid: "93f65129f65f48dcb4f740d3d0cb2d93" member_type: VOTER last_known_addr { host: "127.1.170.129" port: 42153 } } peers { permanent_uuid: "e757ee402fd040fe9a5113b623aad3f2" member_type: VOTER last_known_addr { host: "127.1.170.130" port: 43839 } }
I20260923 20:14:09.784514 22094 consensus_queue.cc:260] T 82056ba9b19543dfa61cb8c444a8eae1 P 4acafeb234cc492e9fcfec9dbf23d23f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4acafeb234cc492e9fcfec9dbf23d23f" member_type: VOTER last_known_addr { host: "127.1.170.131" port: 38677 } } peers { permanent_uuid: "93f65129f65f48dcb4f740d3d0cb2d93" member_type: VOTER last_known_addr { host: "127.1.170.129" port: 42153 } } peers { permanent_uuid: "e757ee402fd040fe9a5113b623aad3f2" member_type: VOTER last_known_addr { host: "127.1.170.130" port: 43839 } }
I20260923 20:14:09.785868 22098 tablet_bootstrap.cc:492] T 82056ba9b19543dfa61cb8c444a8eae1 P e757ee402fd040fe9a5113b623aad3f2: No bootstrap required, opened a new log
I20260923 20:14:09.786123 22095 ts_tablet_manager.cc:1428] T 82056ba9b19543dfa61cb8c444a8eae1 P 93f65129f65f48dcb4f740d3d0cb2d93: Time spent starting tablet: real 0.011s user 0.009s sys 0.002s
I20260923 20:14:09.786242 22098 ts_tablet_manager.cc:1397] T 82056ba9b19543dfa61cb8c444a8eae1 P e757ee402fd040fe9a5113b623aad3f2: Time spent bootstrapping tablet: real 0.011s user 0.009s sys 0.000s
I20260923 20:14:09.794143 22076 heartbeater.cc:503] Master 127.1.170.190:36285 was elected leader, sending a full tablet report...
I20260923 20:14:09.794872 22094 ts_tablet_manager.cc:1428] T 82056ba9b19543dfa61cb8c444a8eae1 P 4acafeb234cc492e9fcfec9dbf23d23f: Time spent starting tablet: real 0.023s user 0.024s sys 0.000s
I20260923 20:14:09.799376 22098 raft_consensus.cc:348] T 82056ba9b19543dfa61cb8c444a8eae1 P e757ee402fd040fe9a5113b623aad3f2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4acafeb234cc492e9fcfec9dbf23d23f" member_type: VOTER last_known_addr { host: "127.1.170.131" port: 38677 } } peers { permanent_uuid: "93f65129f65f48dcb4f740d3d0cb2d93" member_type: VOTER last_known_addr { host: "127.1.170.129" port: 42153 } } peers { permanent_uuid: "e757ee402fd040fe9a5113b623aad3f2" member_type: VOTER last_known_addr { host: "127.1.170.130" port: 43839 } }
I20260923 20:14:09.799751 22098 raft_consensus.cc:374] T 82056ba9b19543dfa61cb8c444a8eae1 P e757ee402fd040fe9a5113b623aad3f2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:14:09.799849 22098 raft_consensus.cc:729] T 82056ba9b19543dfa61cb8c444a8eae1 P e757ee402fd040fe9a5113b623aad3f2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e757ee402fd040fe9a5113b623aad3f2, State: Initialized, Role: FOLLOWER
I20260923 20:14:09.800395 22098 consensus_queue.cc:260] T 82056ba9b19543dfa61cb8c444a8eae1 P e757ee402fd040fe9a5113b623aad3f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4acafeb234cc492e9fcfec9dbf23d23f" member_type: VOTER last_known_addr { host: "127.1.170.131" port: 38677 } } peers { permanent_uuid: "93f65129f65f48dcb4f740d3d0cb2d93" member_type: VOTER last_known_addr { host: "127.1.170.129" port: 42153 } } peers { permanent_uuid: "e757ee402fd040fe9a5113b623aad3f2" member_type: VOTER last_known_addr { host: "127.1.170.130" port: 43839 } }
I20260923 20:14:09.801384 21946 heartbeater.cc:503] Master 127.1.170.190:36285 was elected leader, sending a full tablet report...
I20260923 20:14:09.802106 22098 ts_tablet_manager.cc:1428] T 82056ba9b19543dfa61cb8c444a8eae1 P e757ee402fd040fe9a5113b623aad3f2: Time spent starting tablet: real 0.016s user 0.009s sys 0.004s
I20260923 20:14:09.827826 22100 raft_consensus.cc:482] T 82056ba9b19543dfa61cb8c444a8eae1 P 93f65129f65f48dcb4f740d3d0cb2d93 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 20:14:09.828183 22100 raft_consensus.cc:504] T 82056ba9b19543dfa61cb8c444a8eae1 P 93f65129f65f48dcb4f740d3d0cb2d93 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4acafeb234cc492e9fcfec9dbf23d23f" member_type: VOTER last_known_addr { host: "127.1.170.131" port: 38677 } } peers { permanent_uuid: "93f65129f65f48dcb4f740d3d0cb2d93" member_type: VOTER last_known_addr { host: "127.1.170.129" port: 42153 } } peers { permanent_uuid: "e757ee402fd040fe9a5113b623aad3f2" member_type: VOTER last_known_addr { host: "127.1.170.130" port: 43839 } }
I20260923 20:14:09.830004 22100 leader_election.cc:288] T 82056ba9b19543dfa61cb8c444a8eae1 P 93f65129f65f48dcb4f740d3d0cb2d93 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4acafeb234cc492e9fcfec9dbf23d23f (127.1.170.131:38677), e757ee402fd040fe9a5113b623aad3f2 (127.1.170.130:43839)
I20260923 20:14:09.836644 22089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:14:09.830378 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.170.129:34265 (local address 127.1.170.131:38677)
0923 20:14:09.830657 (+ 279us) server_negotiation.cc:206] Beginning negotiation
0923 20:14:09.830666 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 20:14:09.831019 (+ 353us) server_negotiation.cc:407] Connection header received
0923 20:14:09.831251 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:14:09.831259 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:14:09.831348 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:14:09.831450 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:14:09.832032 (+ 582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:14:09.832783 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:14:09.834673 (+ 1890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:14:09.835625 (+ 952us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:14:09.835809 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:14:09.835878 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0923 20:14:09.836279 (+ 401us) server_negotiation.cc:299] Negotiation successful
0923 20:14:09.836408 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260923 20:14:09.836695 22103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:14:09.830242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.131:38677 (local address 127.1.170.129:34265)
0923 20:14:09.830899 (+ 657us) negotiation.cc:107] Waiting for socket to connect
0923 20:14:09.830916 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0923 20:14:09.831097 (+ 181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:14:09.831476 (+ 379us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:14:09.831480 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:14:09.831516 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:14:09.831878 (+ 362us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:14:09.831887 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:14:09.832925 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:14:09.832933 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:14:09.834464 (+ 1531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:14:09.834476 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:14:09.835890 (+ 1414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:14:09.835898 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:14:09.836076 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:14:09.836111 (+ 35us) client_negotiation.cc:769] Sending connection context
0923 20:14:09.836215 (+ 104us) client_negotiation.cc:240] Negotiation successful
0923 20:14:09.836359 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":522,"thread_start_us":339,"threads_started":1}
I20260923 20:14:09.837687 22093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:14:09.831111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.170.129:37213 (local address 127.1.170.130:43839)
0923 20:14:09.831373 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0923 20:14:09.831382 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 20:14:09.831815 (+ 433us) server_negotiation.cc:407] Connection header received
0923 20:14:09.832022 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:14:09.832027 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:14:09.832104 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:14:09.832201 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:14:09.832734 (+ 533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:14:09.833530 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:14:09.835401 (+ 1871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:14:09.836757 (+ 1356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:14:09.836937 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:14:09.837012 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0923 20:14:09.837310 (+ 298us) server_negotiation.cc:299] Negotiation successful
0923 20:14:09.837444 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260923 20:14:09.837707 22104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:14:09.830984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.130:43839 (local address 127.1.170.129:37213)
0923 20:14:09.831705 (+ 721us) negotiation.cc:107] Waiting for socket to connect
0923 20:14:09.831719 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0923 20:14:09.831871 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:14:09.832232 (+ 361us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:14:09.832236 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:14:09.832259 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:14:09.832567 (+ 308us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:14:09.832576 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:14:09.833698 (+ 1122us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:14:09.833703 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:14:09.835146 (+ 1443us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:14:09.835159 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:14:09.836987 (+ 1828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:14:09.836993 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:14:09.837156 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:14:09.837177 (+ 21us) client_negotiation.cc:769] Sending connection context
0923 20:14:09.837284 (+ 107us) client_negotiation.cc:240] Negotiation successful
0923 20:14:09.837413 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":597,"thread_start_us":380,"threads_started":1}
I20260923 20:14:09.837703 22031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82056ba9b19543dfa61cb8c444a8eae1" candidate_uuid: "93f65129f65f48dcb4f740d3d0cb2d93" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4acafeb234cc492e9fcfec9dbf23d23f" is_pre_election: true
I20260923 20:14:09.838367 22031 raft_consensus.cc:2454] T 82056ba9b19543dfa61cb8c444a8eae1 P 4acafeb234cc492e9fcfec9dbf23d23f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 93f65129f65f48dcb4f740d3d0cb2d93 in term 0.
I20260923 20:14:09.838723 21901 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82056ba9b19543dfa61cb8c444a8eae1" candidate_uuid: "93f65129f65f48dcb4f740d3d0cb2d93" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e757ee402fd040fe9a5113b623aad3f2" is_pre_election: true
I20260923 20:14:09.839092 21703 leader_election.cc:302] T 82056ba9b19543dfa61cb8c444a8eae1 P 93f65129f65f48dcb4f740d3d0cb2d93 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4acafeb234cc492e9fcfec9dbf23d23f, 93f65129f65f48dcb4f740d3d0cb2d93; no voters:
I20260923 20:14:09.839247 21901 raft_consensus.cc:2454] T 82056ba9b19543dfa61cb8c444a8eae1 P e757ee402fd040fe9a5113b623aad3f2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 93f65129f65f48dcb4f740d3d0cb2d93 in term 0.
I20260923 20:14:09.839623 22100 raft_consensus.cc:2781] T 82056ba9b19543dfa61cb8c444a8eae1 P 93f65129f65f48dcb4f740d3d0cb2d93 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 20:14:09.839812 22100 raft_consensus.cc:482] T 82056ba9b19543dfa61cb8c444a8eae1 P 93f65129f65f48dcb4f740d3d0cb2d93 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 20:14:09.839977 22100 raft_consensus.cc:3037] T 82056ba9b19543dfa61cb8c444a8eae1 P 93f65129f65f48dcb4f740d3d0cb2d93 [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:14:09.842680 22100 raft_consensus.cc:504] T 82056ba9b19543dfa61cb8c444a8eae1 P 93f65129f65f48dcb4f740d3d0cb2d93 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4acafeb234cc492e9fcfec9dbf23d23f" member_type: VOTER last_known_addr { host: "127.1.170.131" port: 38677 } } peers { permanent_uuid: "93f65129f65f48dcb4f740d3d0cb2d93" member_type: VOTER last_known_addr { host: "127.1.170.129" port: 42153 } } peers { permanent_uuid: "e757ee402fd040fe9a5113b623aad3f2" member_type: VOTER last_known_addr { host: "127.1.170.130" port: 43839 } }
I20260923 20:14:09.843364 22100 leader_election.cc:288] T 82056ba9b19543dfa61cb8c444a8eae1 P 93f65129f65f48dcb4f740d3d0cb2d93 [CANDIDATE]: Term 1 election: Requested vote from peers 4acafeb234cc492e9fcfec9dbf23d23f (127.1.170.131:38677), e757ee402fd040fe9a5113b623aad3f2 (127.1.170.130:43839)
I20260923 20:14:09.843986 22031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82056ba9b19543dfa61cb8c444a8eae1" candidate_uuid: "93f65129f65f48dcb4f740d3d0cb2d93" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4acafeb234cc492e9fcfec9dbf23d23f"
I20260923 20:14:09.844074 21901 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82056ba9b19543dfa61cb8c444a8eae1" candidate_uuid: "93f65129f65f48dcb4f740d3d0cb2d93" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e757ee402fd040fe9a5113b623aad3f2"
I20260923 20:14:09.844218 22031 raft_consensus.cc:3037] T 82056ba9b19543dfa61cb8c444a8eae1 P 4acafeb234cc492e9fcfec9dbf23d23f [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:14:09.844313 21901 raft_consensus.cc:3037] T 82056ba9b19543dfa61cb8c444a8eae1 P e757ee402fd040fe9a5113b623aad3f2 [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:14:09.846637 21901 raft_consensus.cc:2454] T 82056ba9b19543dfa61cb8c444a8eae1 P e757ee402fd040fe9a5113b623aad3f2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 93f65129f65f48dcb4f740d3d0cb2d93 in term 1.
I20260923 20:14:09.846781 22031 raft_consensus.cc:2454] T 82056ba9b19543dfa61cb8c444a8eae1 P 4acafeb234cc492e9fcfec9dbf23d23f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 93f65129f65f48dcb4f740d3d0cb2d93 in term 1.
I20260923 20:14:09.847085 21703 leader_election.cc:302] T 82056ba9b19543dfa61cb8c444a8eae1 P 93f65129f65f48dcb4f740d3d0cb2d93 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93f65129f65f48dcb4f740d3d0cb2d93, e757ee402fd040fe9a5113b623aad3f2; no voters:
I20260923 20:14:09.847460 22100 raft_consensus.cc:2781] T 82056ba9b19543dfa61cb8c444a8eae1 P 93f65129f65f48dcb4f740d3d0cb2d93 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 20:14:09.847947 22100 raft_consensus.cc:686] T 82056ba9b19543dfa61cb8c444a8eae1 P 93f65129f65f48dcb4f740d3d0cb2d93 [term 1 LEADER]: Becoming Leader. State: Replica: 93f65129f65f48dcb4f740d3d0cb2d93, State: Running, Role: LEADER
I20260923 20:14:09.848516 22100 consensus_queue.cc:237] T 82056ba9b19543dfa61cb8c444a8eae1 P 93f65129f65f48dcb4f740d3d0cb2d93 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4acafeb234cc492e9fcfec9dbf23d23f" member_type: VOTER last_known_addr { host: "127.1.170.131" port: 38677 } } peers { permanent_uuid: "93f65129f65f48dcb4f740d3d0cb2d93" member_type: VOTER last_known_addr { host: "127.1.170.129" port: 42153 } } peers { permanent_uuid: "e757ee402fd040fe9a5113b623aad3f2" member_type: VOTER last_known_addr { host: "127.1.170.130" port: 43839 } }
I20260923 20:14:09.853014 21629 catalog_manager.cc:5495] T 82056ba9b19543dfa61cb8c444a8eae1 P 93f65129f65f48dcb4f740d3d0cb2d93 reported cstate change: term changed from 0 to 1, leader changed from <none> to 93f65129f65f48dcb4f740d3d0cb2d93 (127.1.170.129). New cstate: current_term: 1 leader_uuid: "93f65129f65f48dcb4f740d3d0cb2d93" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4acafeb234cc492e9fcfec9dbf23d23f" member_type: VOTER last_known_addr { host: "127.1.170.131" port: 38677 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "93f65129f65f48dcb4f740d3d0cb2d93" member_type: VOTER last_known_addr { host: "127.1.170.129" port: 42153 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e757ee402fd040fe9a5113b623aad3f2" member_type: VOTER last_known_addr { host: "127.1.170.130" port: 43839 } health_report { overall_health: UNKNOWN } } }
I20260923 20:14:09.872385 22091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:14:09.867077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44570 (local address 127.1.170.129:42153)
0923 20:14:09.867263 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0923 20:14:09.867267 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 20:14:09.867285 (+ 18us) server_negotiation.cc:407] Connection header received
0923 20:14:09.867458 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:14:09.867462 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:14:09.867558 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:14:09.867681 (+ 123us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 20:14:09.868040 (+ 359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:14:09.868733 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:14:09.870280 (+ 1547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:14:09.870730 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:14:09.870833 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:14:09.871264 (+ 431us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 20:14:09.871782 (+ 518us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 20:14:09.871895 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0923 20:14:09.872017 (+ 122us) server_negotiation.cc:299] Negotiation successful
0923 20:14:09.872080 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260923 20:14:09.881000 22031 raft_consensus.cc:1261] T 82056ba9b19543dfa61cb8c444a8eae1 P 4acafeb234cc492e9fcfec9dbf23d23f [term 1 FOLLOWER]: Refusing update from remote peer 93f65129f65f48dcb4f740d3d0cb2d93: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 20:14:09.881001 21901 raft_consensus.cc:1261] T 82056ba9b19543dfa61cb8c444a8eae1 P e757ee402fd040fe9a5113b623aad3f2 [term 1 FOLLOWER]: Refusing update from remote peer 93f65129f65f48dcb4f740d3d0cb2d93: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 20:14:09.882020 22105 consensus_queue.cc:1035] T 82056ba9b19543dfa61cb8c444a8eae1 P 93f65129f65f48dcb4f740d3d0cb2d93 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e757ee402fd040fe9a5113b623aad3f2" member_type: VOTER last_known_addr { host: "127.1.170.130" port: 43839 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 20:14:09.882424 22100 consensus_queue.cc:1035] T 82056ba9b19543dfa61cb8c444a8eae1 P 93f65129f65f48dcb4f740d3d0cb2d93 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4acafeb234cc492e9fcfec9dbf23d23f" member_type: VOTER last_known_addr { host: "127.1.170.131" port: 38677 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 20:14:09.894446 22108 mvcc.cc:205] Tried to move back new op lower bound from 7332636466701684736 to 7332636466583404544. Current Snapshot: MvccSnapshot[applied={T|T < 7332636466701684736}]
I20260923 20:14:09.894755 22110 mvcc.cc:205] Tried to move back new op lower bound from 7332636466701684736 to 7332636466583404544. Current Snapshot: MvccSnapshot[applied={T|T < 7332636466701684736}]
I20260923 20:14:09.910209 21629 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50042:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 20:14:09.910485 21629 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50042:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 20:14:09.914563 21629 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 83401988a73c41d68a46098bb1023fe2: Sending DeleteTablet for 3 replicas of tablet 82056ba9b19543dfa61cb8c444a8eae1
I20260923 20:14:09.915942 22011 tablet_service.cc:1514] Processing DeleteTablet for tablet 82056ba9b19543dfa61cb8c444a8eae1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 20:14:09 UTC) from {username='slave'} at 127.0.0.1:35840
I20260923 20:14:09.915949 21750 tablet_service.cc:1514] Processing DeleteTablet for tablet 82056ba9b19543dfa61cb8c444a8eae1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 20:14:09 UTC) from {username='slave'} at 127.0.0.1:44562
I20260923 20:14:09.916913 21881 tablet_service.cc:1514] Processing DeleteTablet for tablet 82056ba9b19543dfa61cb8c444a8eae1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 20:14:09 UTC) from {username='slave'} at 127.0.0.1:41398
I20260923 20:14:09.917663 1706 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskqUuku6/build/asan/bin/kudu pid 21688
I20260923 20:14:09.928818 22117 tablet_replica.cc:330] T 82056ba9b19543dfa61cb8c444a8eae1 P 4acafeb234cc492e9fcfec9dbf23d23f: stopping tablet replica
I20260923 20:14:09.928905 22118 tablet_replica.cc:330] T 82056ba9b19543dfa61cb8c444a8eae1 P 93f65129f65f48dcb4f740d3d0cb2d93: stopping tablet replica
I20260923 20:14:09.929369 22117 raft_consensus.cc:2229] T 82056ba9b19543dfa61cb8c444a8eae1 P 4acafeb234cc492e9fcfec9dbf23d23f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 20:14:09.929802 22118 raft_consensus.cc:2229] T 82056ba9b19543dfa61cb8c444a8eae1 P 93f65129f65f48dcb4f740d3d0cb2d93 [term 1 LEADER]: Raft consensus shutting down.
I20260923 20:14:09.930143 22117 raft_consensus.cc:2258] T 82056ba9b19543dfa61cb8c444a8eae1 P 4acafeb234cc492e9fcfec9dbf23d23f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 20:14:09.930994 22118 raft_consensus.cc:2258] T 82056ba9b19543dfa61cb8c444a8eae1 P 93f65129f65f48dcb4f740d3d0cb2d93 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 20:14:09.933058 22117 ts_tablet_manager.cc:1905] T 82056ba9b19543dfa61cb8c444a8eae1 P 4acafeb234cc492e9fcfec9dbf23d23f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 20:14:09.933459 22118 ts_tablet_manager.cc:1905] T 82056ba9b19543dfa61cb8c444a8eae1 P 93f65129f65f48dcb4f740d3d0cb2d93: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 20:14:09.941620 22118 ts_tablet_manager.cc:1918] T 82056ba9b19543dfa61cb8c444a8eae1 P 93f65129f65f48dcb4f740d3d0cb2d93: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 20:14:09.941979 22118 log.cc:1196] T 82056ba9b19543dfa61cb8c444a8eae1 P 93f65129f65f48dcb4f740d3d0cb2d93: Deleting WAL directory at /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790194121577027-1706-0/minicluster-data/ts-0/wal/wals/82056ba9b19543dfa61cb8c444a8eae1
I20260923 20:14:09.942229 22117 ts_tablet_manager.cc:1918] T 82056ba9b19543dfa61cb8c444a8eae1 P 4acafeb234cc492e9fcfec9dbf23d23f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 20:14:09.942443 22117 log.cc:1196] T 82056ba9b19543dfa61cb8c444a8eae1 P 4acafeb234cc492e9fcfec9dbf23d23f: Deleting WAL directory at /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790194121577027-1706-0/minicluster-data/ts-2/wal/wals/82056ba9b19543dfa61cb8c444a8eae1
I20260923 20:14:09.942796 22118 ts_tablet_manager.cc:1939] T 82056ba9b19543dfa61cb8c444a8eae1 P 93f65129f65f48dcb4f740d3d0cb2d93: Deleting consensus metadata
I20260923 20:14:09.943146 22117 ts_tablet_manager.cc:1939] T 82056ba9b19543dfa61cb8c444a8eae1 P 4acafeb234cc492e9fcfec9dbf23d23f: Deleting consensus metadata
I20260923 20:14:09.945055 21613 catalog_manager.cc:4841] TS 4acafeb234cc492e9fcfec9dbf23d23f (127.1.170.131:38677): tablet 82056ba9b19543dfa61cb8c444a8eae1 (table test-table [id=edc1ae7cb17c40889f6492471d47a112]) successfully deleted
I20260923 20:14:09.945374 21613 catalog_manager.cc:4841] TS 93f65129f65f48dcb4f740d3d0cb2d93 (127.1.170.129:42153): tablet 82056ba9b19543dfa61cb8c444a8eae1 (table test-table [id=edc1ae7cb17c40889f6492471d47a112]) successfully deleted
I20260923 20:14:09.946385 22119 tablet_replica.cc:330] T 82056ba9b19543dfa61cb8c444a8eae1 P e757ee402fd040fe9a5113b623aad3f2: stopping tablet replica
I20260923 20:14:09.946869 22119 raft_consensus.cc:2229] T 82056ba9b19543dfa61cb8c444a8eae1 P e757ee402fd040fe9a5113b623aad3f2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 20:14:09.947336 22119 raft_consensus.cc:2258] T 82056ba9b19543dfa61cb8c444a8eae1 P e757ee402fd040fe9a5113b623aad3f2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 20:14:09.949360 22119 ts_tablet_manager.cc:1905] T 82056ba9b19543dfa61cb8c444a8eae1 P e757ee402fd040fe9a5113b623aad3f2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 20:14:09.952149 22091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:14:09.919454 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44574 (local address 127.1.170.129:42153)
0923 20:14:09.919636 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0923 20:14:09.919645 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 20:14:09.943761 (+ 24116us) server_negotiation.cc:407] Connection header received
0923 20:14:09.944306 (+ 545us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:14:09.944312 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:14:09.944419 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:14:09.944500 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:14:09.945203 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:14:09.946637 (+ 1434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:14:09.948500 (+ 1863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:14:09.949120 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:14:09.949260 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:14:09.950483 (+ 1223us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:14:09.950493 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:14:09.950509 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 20:14:09.951287 (+ 778us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:14:09.951349 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 20:14:09.951373 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:14:09.951485 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0923 20:14:09.951653 (+ 168us) server_negotiation.cc:299] Negotiation successful
0923 20:14:09.951771 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260923 20:14:09.953049 21810 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:14:09.957365 22119 ts_tablet_manager.cc:1918] T 82056ba9b19543dfa61cb8c444a8eae1 P e757ee402fd040fe9a5113b623aad3f2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 20:14:09.957538 22119 log.cc:1196] T 82056ba9b19543dfa61cb8c444a8eae1 P e757ee402fd040fe9a5113b623aad3f2: Deleting WAL directory at /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790194121577027-1706-0/minicluster-data/ts-1/wal/wals/82056ba9b19543dfa61cb8c444a8eae1
I20260923 20:14:09.958066 22119 ts_tablet_manager.cc:1939] T 82056ba9b19543dfa61cb8c444a8eae1 P e757ee402fd040fe9a5113b623aad3f2: Deleting consensus metadata
I20260923 20:14:09.959785 21613 catalog_manager.cc:4841] TS e757ee402fd040fe9a5113b623aad3f2 (127.1.170.130:43839): tablet 82056ba9b19543dfa61cb8c444a8eae1 (table test-table [id=edc1ae7cb17c40889f6492471d47a112]) successfully deleted
I20260923 20:14:10.052815 1706 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqUuku6/build/asan/bin/kudu with pid 21688
I20260923 20:14:10.077747 1706 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskqUuku6/build/asan/bin/kudu pid 21819
I20260923 20:14:10.085935 22093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:14:10.078605 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41412 (local address 127.1.170.130:43839)
0923 20:14:10.078781 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0923 20:14:10.078786 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 20:14:10.078805 (+ 19us) server_negotiation.cc:407] Connection header received
0923 20:14:10.079006 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:14:10.079010 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:14:10.079112 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:14:10.079205 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:14:10.079671 (+ 466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:14:10.080451 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:14:10.082485 (+ 2034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:14:10.082939 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:14:10.083053 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:14:10.084285 (+ 1232us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:14:10.084294 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:14:10.084309 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 20:14:10.084952 (+ 643us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:14:10.085007 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 20:14:10.085027 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:14:10.085123 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0923 20:14:10.085339 (+ 216us) server_negotiation.cc:299] Negotiation successful
0923 20:14:10.085437 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260923 20:14:10.086647 21941 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:14:10.176143 1706 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqUuku6/build/asan/bin/kudu with pid 21819
I20260923 20:14:10.193089 1706 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskqUuku6/build/asan/bin/kudu pid 21949
I20260923 20:14:10.199757 22089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:14:10.193844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35846 (local address 127.1.170.131:38677)
0923 20:14:10.194066 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0923 20:14:10.194070 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 20:14:10.194176 (+ 106us) server_negotiation.cc:407] Connection header received
0923 20:14:10.194320 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:14:10.194324 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:14:10.194411 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:14:10.194495 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:14:10.194865 (+ 370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:14:10.195585 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:14:10.196788 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:14:10.197168 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:14:10.197261 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:14:10.198374 (+ 1113us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:14:10.198383 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:14:10.198398 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 20:14:10.198983 (+ 585us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:14:10.199029 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 20:14:10.199048 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:14:10.199143 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0923 20:14:10.199276 (+ 133us) server_negotiation.cc:299] Negotiation successful
0923 20:14:10.199365 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109}
I20260923 20:14:10.200413 22071 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:14:10.289149 1706 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqUuku6/build/asan/bin/kudu with pid 21949
I20260923 20:14:10.307752 1706 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskqUuku6/build/asan/bin/kudu pid 21598
I20260923 20:14:10.308851 21659 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:14:10.419312 1706 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqUuku6/build/asan/bin/kudu with pid 21598
2026-09-23T20:14:10Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (2936 ms)
[----------] 6 tests from EncryptionPolicyTest (22223 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790194121577027-1706-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 20:14:10 dist-test-slave-th4g krb5kdc[22127](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 20:14:10 dist-test-slave-th4g krb5kdc[22127](info): set up 2 sockets
Sep 23 20:14:10 dist-test-slave-th4g krb5kdc[22127](info): commencing operation
krb5kdc: starting...
W20260923 20:14:11.554644 1706 mini_kdc.cc:121] Time spent starting KDC: real 1.088s user 0.005s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790194121577027-1706-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790194121577027-1706-0/krb5kdc/test-user.keytab.
Sep 23 20:14:11 dist-test-slave-th4g krb5kdc[22127](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194451, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T20:14:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T20:14:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790194121577027-1706-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790194121577027-1706-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260923 20:14:11.830678 1706 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqUuku6/build/asan/bin/kudu
/tmp/dist-test-taskqUuku6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790194121577027-1706-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790194121577027-1706-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790194121577027-1706-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790194121577027-1706-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44065
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790194121577027-1706-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36905
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44065
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790194121577027-1706-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790194121577027-1706-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790194121577027-1706-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:14:12.176069 22140 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:14:12.176374 22140 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:14:12.176524 22140 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:14:12.184018 22140 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 20:14:12.184152 22140 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:14:12.184242 22140 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:14:12.184331 22140 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 20:14:12.184394 22140 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 20:14:12.195631 22140 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36905
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790194121577027-1706-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790194121577027-1706-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44065
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790194121577027-1706-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44065
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790194121577027-1706-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790194121577027-1706-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:14:12.197045 22140 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:14:12.198391 22140 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:14:12.211916 22145 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:14:12.211881 22146 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:14:12.211915 22148 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:14:12.213061 22140 server_base.cc:1034] running on GCE node
I20260923 20:14:12.214359 22140 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:14:12.216146 22140 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:14:12.217382 22140 hybrid_clock.cc:648] HybridClock initialized: now 1790194452217314 us; error 54 us; skew 500 ppm
I20260923 20:14:12.217799 22140 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:14:12 dist-test-slave-th4g krb5kdc[22127](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194452, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:14:12.227952 22140 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260923 20:14:12.229743 22140 webserver.cc:460] Webserver started at http://127.0.0.1:35765/ using document root <none> and password file <none>
I20260923 20:14:12.230562 22140 fs_manager.cc:359] Metadata directory not provided
I20260923 20:14:12.230693 22140 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:14:12.231032 22140 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:14:12.234496 22140 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790194121577027-1706-0/minicluster-data/master-0/data/instance:
uuid: "785fb5118711489d926bea0ba104aabc"
format_stamp: "Formatted at 2026-09-23 20:14:12 on dist-test-slave-th4g"
server_key: "1cfef2d4653d41c12e9bd89149f3ca9d"
server_key_iv: "b6ebf36580e500a460e2d5283db3afa2"
server_key_version: "encryptionkey@0"
I20260923 20:14:12.235350 22140 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790194121577027-1706-0/minicluster-data/master-0/wal/instance:
uuid: "785fb5118711489d926bea0ba104aabc"
format_stamp: "Formatted at 2026-09-23 20:14:12 on dist-test-slave-th4g"
server_key: "1cfef2d4653d41c12e9bd89149f3ca9d"
server_key_iv: "b6ebf36580e500a460e2d5283db3afa2"
server_key_version: "encryptionkey@0"
I20260923 20:14:12.240841 22140 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260923 20:14:12.244720 22155 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:14:12.246294 22140 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 20:14:12.246492 22140 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790194121577027-1706-0/minicluster-data/master-0/data,/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790194121577027-1706-0/minicluster-data/master-0/wal
uuid: "785fb5118711489d926bea0ba104aabc"
format_stamp: "Formatted at 2026-09-23 20:14:12 on dist-test-slave-th4g"
server_key: "1cfef2d4653d41c12e9bd89149f3ca9d"
server_key_iv: "b6ebf36580e500a460e2d5283db3afa2"
server_key_version: "encryptionkey@0"
I20260923 20:14:12.246711 22140 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790194121577027-1706-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790194121577027-1706-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790194121577027-1706-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:14:12.267685 22140 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:14:12.272279 22140 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:14:12.272593 22140 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:14:12.292371 22140 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44065
I20260923 20:14:12.292418 22206 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44065 every 8 connection(s)
I20260923 20:14:12.293833 22140 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790194121577027-1706-0/minicluster-data/master-0/data/info.pb
I20260923 20:14:12.297679 22207 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:14:12.297710 1706 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqUuku6/build/asan/bin/kudu as pid 22140
I20260923 20:14:12.297935 1706 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790194121577027-1706-0/minicluster-data/master-0/wal/instance
I20260923 20:14:12.298748 1706 external_mini_cluster.cc:1428] Setting key 36d4d8fe4f176beb04b1f2bb63d9e0b7
I20260923 20:14:12.312139 22207 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 785fb5118711489d926bea0ba104aabc: Bootstrap starting.
Sep 23 20:14:12 dist-test-slave-th4g krb5kdc[22127](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194451, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260923 20:14:12.316459 22207 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 785fb5118711489d926bea0ba104aabc: Neither blocks nor log segments found. Creating new log.
I20260923 20:14:12.317757 22207 log.cc:824] T 00000000000000000000000000000000 P 785fb5118711489d926bea0ba104aabc: Log is configured to *not* fsync() on all Append() calls
I20260923 20:14:12.321481 22207 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 785fb5118711489d926bea0ba104aabc: No bootstrap required, opened a new log
I20260923 20:14:12.325850 22210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:14:12.300869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34505 (local address 127.0.0.1:44065)
0923 20:14:12.301677 (+ 808us) server_negotiation.cc:206] Beginning negotiation
0923 20:14:12.301690 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0923 20:14:12.301736 (+ 46us) server_negotiation.cc:407] Connection header received
0923 20:14:12.303983 (+ 2247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:14:12.304010 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:14:12.304445 (+ 435us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:14:12.304782 (+ 337us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:14:12.306568 (+ 1786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:14:12.307877 (+ 1309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:14:12.310009 (+ 2132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:14:12.310875 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:14:12.311044 (+ 169us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:14:12.316487 (+ 5443us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:14:12.316496 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:14:12.316512 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:14:12.316564 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:14:12.320379 (+ 3815us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:14:12.321689 (+ 1310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:14:12.321698 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:14:12.321708 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:14:12.321957 (+ 249us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:14:12.322587 (+ 630us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:14:12.322591 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:14:12.322593 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:14:12.323166 (+ 573us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:14:12.323606 (+ 440us) server_negotiation.cc:1035] Waiting for connection context
0923 20:14:12.324159 (+ 553us) server_negotiation.cc:299] Negotiation successful
0923 20:14:12.324719 (+ 560us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":611,"thread_start_us":391,"threads_started":1}
I20260923 20:14:12.330437 1706 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 20:14:12.331023 22207 raft_consensus.cc:348] T 00000000000000000000000000000000 P 785fb5118711489d926bea0ba104aabc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "785fb5118711489d926bea0ba104aabc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44065 } }
I20260923 20:14:12.331482 22207 raft_consensus.cc:374] T 00000000000000000000000000000000 P 785fb5118711489d926bea0ba104aabc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:14:12.331578 22207 raft_consensus.cc:729] T 00000000000000000000000000000000 P 785fb5118711489d926bea0ba104aabc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 785fb5118711489d926bea0ba104aabc, State: Initialized, Role: FOLLOWER
I20260923 20:14:12.332237 22207 consensus_queue.cc:260] T 00000000000000000000000000000000 P 785fb5118711489d926bea0ba104aabc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "785fb5118711489d926bea0ba104aabc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44065 } }
I20260923 20:14:12.332513 22207 raft_consensus.cc:388] T 00000000000000000000000000000000 P 785fb5118711489d926bea0ba104aabc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 20:14:12.332638 22207 raft_consensus.cc:482] T 00000000000000000000000000000000 P 785fb5118711489d926bea0ba104aabc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 20:14:12.332824 22207 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 785fb5118711489d926bea0ba104aabc [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:14:12.336419 22207 raft_consensus.cc:504] T 00000000000000000000000000000000 P 785fb5118711489d926bea0ba104aabc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "785fb5118711489d926bea0ba104aabc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44065 } }
I20260923 20:14:12.337059 22207 leader_election.cc:302] T 00000000000000000000000000000000 P 785fb5118711489d926bea0ba104aabc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 785fb5118711489d926bea0ba104aabc; no voters:
I20260923 20:14:12.337814 22207 leader_election.cc:288] T 00000000000000000000000000000000 P 785fb5118711489d926bea0ba104aabc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 20:14:12.338011 22217 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 785fb5118711489d926bea0ba104aabc [term 1 FOLLOWER]: Leader election won for term 1
I20260923 20:14:12.340180 22207 sys_catalog.cc:564] T 00000000000000000000000000000000 P 785fb5118711489d926bea0ba104aabc [sys.catalog]: configured and running, proceeding with master startup.
I20260923 20:14:12.340389 22217 raft_consensus.cc:686] T 00000000000000000000000000000000 P 785fb5118711489d926bea0ba104aabc [term 1 LEADER]: Becoming Leader. State: Replica: 785fb5118711489d926bea0ba104aabc, State: Running, Role: LEADER
I20260923 20:14:12.341259 22217 consensus_queue.cc:237] T 00000000000000000000000000000000 P 785fb5118711489d926bea0ba104aabc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "785fb5118711489d926bea0ba104aabc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44065 } }
I20260923 20:14:12.344904 22218 sys_catalog.cc:455] T 00000000000000000000000000000000 P 785fb5118711489d926bea0ba104aabc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "785fb5118711489d926bea0ba104aabc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "785fb5118711489d926bea0ba104aabc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44065 } } }
I20260923 20:14:12.346143 22218 sys_catalog.cc:458] T 00000000000000000000000000000000 P 785fb5118711489d926bea0ba104aabc [sys.catalog]: This master's current role is: LEADER
I20260923 20:14:12.345743 22219 sys_catalog.cc:455] T 00000000000000000000000000000000 P 785fb5118711489d926bea0ba104aabc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 785fb5118711489d926bea0ba104aabc. Latest consensus state: current_term: 1 leader_uuid: "785fb5118711489d926bea0ba104aabc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "785fb5118711489d926bea0ba104aabc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44065 } } }
I20260923 20:14:12.346447 22219 sys_catalog.cc:458] T 00000000000000000000000000000000 P 785fb5118711489d926bea0ba104aabc [sys.catalog]: This master's current role is: LEADER
I20260923 20:14:12.353552 22227 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 20:14:12.359290 22227 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 20:14:12.359539 22210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:14:12.336381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43805 (local address 127.0.0.1:44065)
0923 20:14:12.336595 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0923 20:14:12.336604 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 20:14:12.337116 (+ 512us) server_negotiation.cc:407] Connection header received
0923 20:14:12.337265 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:14:12.337270 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:14:12.337352 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:14:12.337469 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:14:12.340135 (+ 2666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:14:12.341752 (+ 1617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:14:12.343908 (+ 2156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:14:12.345323 (+ 1415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:14:12.345540 (+ 217us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:14:12.350635 (+ 5095us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:14:12.350643 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:14:12.350648 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:14:12.350693 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:14:12.355902 (+ 5209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:14:12.356813 (+ 911us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:14:12.356819 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:14:12.356822 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:14:12.356983 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:14:12.357462 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:14:12.357467 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:14:12.357471 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:14:12.357949 (+ 478us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:14:12.358231 (+ 282us) server_negotiation.cc:1035] Waiting for connection context
0923 20:14:12.358647 (+ 416us) server_negotiation.cc:299] Negotiation successful
0923 20:14:12.358826 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260923 20:14:12.370332 22227 catalog_manager.cc:1347] Generated new cluster ID: 6a7e2902079b4f9a8975cdf7a9485777
I20260923 20:14:12.370520 22227 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 20:14:12.390035 22227 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 20:14:12.391999 22227 catalog_manager.cc:1504] Loading token signing keys...
I20260923 20:14:12.399765 22227 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 785fb5118711489d926bea0ba104aabc: Generated new TSK 0
I20260923 20:14:12.400724 22227 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 20:14:12.413982 1706 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskqUuku6/build/asan/bin/kudu pid 22140
I20260923 20:14:12.415484 22202 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:14:12.512190 1706 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqUuku6/build/asan/bin/kudu with pid 22140
W20260923 20:14:12.530229 22125 connection.cc:535] client connection to 127.0.0.1:44065 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:44065): Connection reset by peer (error 104)
2026-09-23T20:14:12Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (2083 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790194121577027-1706-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 20:14:12 dist-test-slave-th4g krb5kdc[22239](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 20:14:12 dist-test-slave-th4g krb5kdc[22239](info): set up 2 sockets
Sep 23 20:14:12 dist-test-slave-th4g krb5kdc[22239](info): commencing operation
krb5kdc: starting...
W20260923 20:14:13.662380 1706 mini_kdc.cc:121] Time spent starting KDC: real 1.113s user 0.006s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790194121577027-1706-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790194121577027-1706-0/krb5kdc/test-user.keytab.
Sep 23 20:14:13 dist-test-slave-th4g krb5kdc[22239](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194453, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T20:14:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T20:14:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790194121577027-1706-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790194121577027-1706-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260923 20:14:13.926970 1706 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqUuku6/build/asan/bin/kudu
/tmp/dist-test-taskqUuku6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790194121577027-1706-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790194121577027-1706-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790194121577027-1706-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790194121577027-1706-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36561
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790194121577027-1706-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40553
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36561
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790194121577027-1706-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790194121577027-1706-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790194121577027-1706-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:14:14.255252 22252 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:14:14.255618 22252 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:14:14.255808 22252 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:14:14.263001 22252 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 20:14:14.263160 22252 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:14:14.263276 22252 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:14:14.263387 22252 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 20:14:14.263476 22252 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 20:14:14.273448 22252 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40553
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790194121577027-1706-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790194121577027-1706-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36561
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790194121577027-1706-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36561
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790194121577027-1706-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790194121577027-1706-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:14:14.274771 22252 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:14:14.276072 22252 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:14:14.287959 22260 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:14:14.288057 22258 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:14:14.288029 22257 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:14:14.288800 22252 server_base.cc:1034] running on GCE node
I20260923 20:14:14.289568 22252 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:14:14.291133 22252 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:14:14.292377 22252 hybrid_clock.cc:648] HybridClock initialized: now 1790194454292320 us; error 44 us; skew 500 ppm
I20260923 20:14:14.292793 22252 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:14:14 dist-test-slave-th4g krb5kdc[22239](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194454, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:14:14.296022 22252 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260923 20:14:14.297564 22252 webserver.cc:460] Webserver started at http://127.0.0.1:41241/ using document root <none> and password file <none>
I20260923 20:14:14.298341 22252 fs_manager.cc:359] Metadata directory not provided
I20260923 20:14:14.298485 22252 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:14:14.298790 22252 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:14:14.301663 22252 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790194121577027-1706-0/minicluster-data/master-0/data/instance:
uuid: "9d237e218fa74ae28abb966aab2f1d54"
format_stamp: "Formatted at 2026-09-23 20:14:14 on dist-test-slave-th4g"
server_key: "3094dd85d93a1e5018c849358baba567"
server_key_iv: "9605b72e7f3cc9868180566d37f7e57d"
server_key_version: "encryptionkey@0"
I20260923 20:14:14.302484 22252 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790194121577027-1706-0/minicluster-data/master-0/wal/instance:
uuid: "9d237e218fa74ae28abb966aab2f1d54"
format_stamp: "Formatted at 2026-09-23 20:14:14 on dist-test-slave-th4g"
server_key: "3094dd85d93a1e5018c849358baba567"
server_key_iv: "9605b72e7f3cc9868180566d37f7e57d"
server_key_version: "encryptionkey@0"
I20260923 20:14:14.307893 22252 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260923 20:14:14.311317 22267 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:14:14.312750 22252 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260923 20:14:14.313009 22252 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790194121577027-1706-0/minicluster-data/master-0/data,/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790194121577027-1706-0/minicluster-data/master-0/wal
uuid: "9d237e218fa74ae28abb966aab2f1d54"
format_stamp: "Formatted at 2026-09-23 20:14:14 on dist-test-slave-th4g"
server_key: "3094dd85d93a1e5018c849358baba567"
server_key_iv: "9605b72e7f3cc9868180566d37f7e57d"
server_key_version: "encryptionkey@0"
I20260923 20:14:14.313239 22252 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790194121577027-1706-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790194121577027-1706-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790194121577027-1706-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:14:14.326289 22252 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:14:14.331034 22252 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:14:14.331346 22252 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:14:14.351842 22252 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36561
I20260923 20:14:14.351874 22318 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36561 every 8 connection(s)
I20260923 20:14:14.353343 22252 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790194121577027-1706-0/minicluster-data/master-0/data/info.pb
I20260923 20:14:14.357065 22319 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:14:14.361866 1706 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqUuku6/build/asan/bin/kudu as pid 22252
I20260923 20:14:14.362162 1706 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790194121577027-1706-0/minicluster-data/master-0/wal/instance
I20260923 20:14:14.362613 1706 external_mini_cluster.cc:1428] Setting key 1abef7aff310347a32e2631fa1818f4d
I20260923 20:14:14.368232 22319 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9d237e218fa74ae28abb966aab2f1d54: Bootstrap starting.
I20260923 20:14:14.372362 22319 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9d237e218fa74ae28abb966aab2f1d54: Neither blocks nor log segments found. Creating new log.
I20260923 20:14:14.373551 22319 log.cc:824] T 00000000000000000000000000000000 P 9d237e218fa74ae28abb966aab2f1d54: Log is configured to *not* fsync() on all Append() calls
I20260923 20:14:14.376899 22319 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9d237e218fa74ae28abb966aab2f1d54: No bootstrap required, opened a new log
I20260923 20:14:14.377744 22322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:14:14.364864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58875 (local address 127.0.0.1:36561)
0923 20:14:14.365755 (+ 891us) server_negotiation.cc:206] Beginning negotiation
0923 20:14:14.365768 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0923 20:14:14.365811 (+ 43us) server_negotiation.cc:407] Connection header received
0923 20:14:14.367709 (+ 1898us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:14:14.367731 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:14:14.368147 (+ 416us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:14:14.368552 (+ 405us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:14:14.369120 (+ 568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:14:14.370823 (+ 1703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:14:14.372718 (+ 1895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:14:14.373508 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:14:14.373655 (+ 147us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:14:14.375164 (+ 1509us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:14:14.375172 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:14:14.375187 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 20:14:14.375736 (+ 549us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:14:14.375821 (+ 85us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 20:14:14.375847 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:14:14.375986 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0923 20:14:14.376431 (+ 445us) server_negotiation.cc:299] Negotiation successful
0923 20:14:14.376634 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":640,"thread_start_us":411,"threads_started":1}
I20260923 20:14:14.382038 1706 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 20:14:14.386706 22319 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9d237e218fa74ae28abb966aab2f1d54 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d237e218fa74ae28abb966aab2f1d54" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36561 } }
I20260923 20:14:14.387256 22319 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9d237e218fa74ae28abb966aab2f1d54 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:14:14.387389 22319 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9d237e218fa74ae28abb966aab2f1d54 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d237e218fa74ae28abb966aab2f1d54, State: Initialized, Role: FOLLOWER
I20260923 20:14:14.388185 22319 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9d237e218fa74ae28abb966aab2f1d54 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d237e218fa74ae28abb966aab2f1d54" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36561 } }
I20260923 20:14:14.388527 22319 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9d237e218fa74ae28abb966aab2f1d54 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 20:14:14.388690 22319 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9d237e218fa74ae28abb966aab2f1d54 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 20:14:14.388891 22319 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9d237e218fa74ae28abb966aab2f1d54 [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:14:14.391299 22319 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9d237e218fa74ae28abb966aab2f1d54 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d237e218fa74ae28abb966aab2f1d54" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36561 } }
I20260923 20:14:14.391835 22319 leader_election.cc:302] T 00000000000000000000000000000000 P 9d237e218fa74ae28abb966aab2f1d54 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9d237e218fa74ae28abb966aab2f1d54; no voters:
I20260923 20:14:14.392549 22319 leader_election.cc:288] T 00000000000000000000000000000000 P 9d237e218fa74ae28abb966aab2f1d54 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 20:14:14.392691 22329 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9d237e218fa74ae28abb966aab2f1d54 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 20:14:14.394939 22322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:14:14.387419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36331 (local address 127.0.0.1:36561)
0923 20:14:14.387652 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0923 20:14:14.387659 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:14:14.388129 (+ 470us) server_negotiation.cc:407] Connection header received
0923 20:14:14.388309 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:14:14.388313 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:14:14.388398 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:14:14.388547 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:14:14.389208 (+ 661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:14:14.390206 (+ 998us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:14:14.391704 (+ 1498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:14:14.392283 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:14:14.392465 (+ 182us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:14:14.393960 (+ 1495us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:14:14.393967 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:14:14.393972 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 20:14:14.393999 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:14:14.394048 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 20:14:14.394072 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:14:14.394155 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0923 20:14:14.394491 (+ 336us) server_negotiation.cc:299] Negotiation successful
0923 20:14:14.394549 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260923 20:14:14.395720 22319 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9d237e218fa74ae28abb966aab2f1d54 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 20:14:14.398196 22329 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9d237e218fa74ae28abb966aab2f1d54 [term 1 LEADER]: Becoming Leader. State: Replica: 9d237e218fa74ae28abb966aab2f1d54, State: Running, Role: LEADER
I20260923 20:14:14.399034 22329 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9d237e218fa74ae28abb966aab2f1d54 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d237e218fa74ae28abb966aab2f1d54" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36561 } }
I20260923 20:14:14.403147 22331 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9d237e218fa74ae28abb966aab2f1d54 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9d237e218fa74ae28abb966aab2f1d54. Latest consensus state: current_term: 1 leader_uuid: "9d237e218fa74ae28abb966aab2f1d54" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d237e218fa74ae28abb966aab2f1d54" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36561 } } }
I20260923 20:14:14.402999 22330 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9d237e218fa74ae28abb966aab2f1d54 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9d237e218fa74ae28abb966aab2f1d54" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d237e218fa74ae28abb966aab2f1d54" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36561 } } }
I20260923 20:14:14.403463 22330 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9d237e218fa74ae28abb966aab2f1d54 [sys.catalog]: This master's current role is: LEADER
I20260923 20:14:14.403685 22343 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260923 20:14:14.404148 22331 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9d237e218fa74ae28abb966aab2f1d54 [sys.catalog]: This master's current role is: LEADER
I20260923 20:14:14.404302 22344 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 20:14:14.408634 22344 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 20:14:14.417841 22344 catalog_manager.cc:1347] Generated new cluster ID: dbca0bc59b214ebcbfdadd32bec4d319
I20260923 20:14:14.418088 22344 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 20:14:14.446466 22344 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 20:14:14.448377 22344 catalog_manager.cc:1504] Loading token signing keys...
I20260923 20:14:14.462410 22344 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9d237e218fa74ae28abb966aab2f1d54: Generated new TSK 0
I20260923 20:14:14.463505 22344 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 20:14:14.470062 1706 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskqUuku6/build/asan/bin/kudu pid 22252
W20260923 20:14:14.471338 22314 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:58875
W20260923 20:14:14.472182 1706 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskqUuku6/build/asan/bin/kudu pid 22252: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260923 20:14:14.472371 1706 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqUuku6/build/asan/bin/kudu with pid 22252
W20260923 20:14:14.486562 22237 connection.cc:535] client connection to 127.0.0.1:36561 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:36561): Connection reset by peer (error 104)
2026-09-23T20:14:14Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1955 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790194121577027-1706-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 20:14:14 dist-test-slave-th4g krb5kdc[22350](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 20:14:14 dist-test-slave-th4g krb5kdc[22350](info): set up 2 sockets
Sep 23 20:14:14 dist-test-slave-th4g krb5kdc[22350](info): commencing operation
krb5kdc: starting...
W20260923 20:14:15.610450 1706 mini_kdc.cc:121] Time spent starting KDC: real 1.105s user 0.005s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790194121577027-1706-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790194121577027-1706-0/krb5kdc/test-user.keytab.
Sep 23 20:14:15 dist-test-slave-th4g krb5kdc[22350](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194455, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T20:14:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T20:14:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790194121577027-1706-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790194121577027-1706-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260923 20:14:15.879765 1706 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqUuku6/build/asan/bin/kudu
/tmp/dist-test-taskqUuku6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790194121577027-1706-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790194121577027-1706-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790194121577027-1706-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790194121577027-1706-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42325
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790194121577027-1706-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45585
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42325
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790194121577027-1706-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790194121577027-1706-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790194121577027-1706-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:14:16.235579 22363 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:14:16.235879 22363 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:14:16.235987 22363 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:14:16.243503 22363 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 20:14:16.243644 22363 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:14:16.243742 22363 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:14:16.243836 22363 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 20:14:16.243934 22363 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 20:14:16.254944 22363 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45585
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790194121577027-1706-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790194121577027-1706-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42325
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790194121577027-1706-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42325
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790194121577027-1706-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790194121577027-1706-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:14:16.256458 22363 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:14:16.257915 22363 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:14:16.271517 22368 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:14:16.271543 22369 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:14:16.271543 22371 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:14:16.272680 22363 server_base.cc:1034] running on GCE node
I20260923 20:14:16.273607 22363 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:14:16.275215 22363 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:14:16.276521 22363 hybrid_clock.cc:648] HybridClock initialized: now 1790194456276459 us; error 59 us; skew 500 ppm
I20260923 20:14:16.276973 22363 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:14:16 dist-test-slave-th4g krb5kdc[22350](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194456, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:14:16.280750 22363 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260923 20:14:16.282598 22363 webserver.cc:460] Webserver started at http://127.0.0.1:44055/ using document root <none> and password file <none>
I20260923 20:14:16.283375 22363 fs_manager.cc:359] Metadata directory not provided
I20260923 20:14:16.283542 22363 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:14:16.283895 22363 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:14:16.287503 22363 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790194121577027-1706-0/minicluster-data/master-0/data/instance:
uuid: "a5b935068d21402a821b68077420d1a9"
format_stamp: "Formatted at 2026-09-23 20:14:16 on dist-test-slave-th4g"
server_key: "1ef29069ca80bd721ec1ea25f608d35f"
server_key_iv: "1d4e3898150fcf5b1f0bd427dd889cdd"
server_key_version: "encryptionkey@0"
I20260923 20:14:16.288944 22363 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790194121577027-1706-0/minicluster-data/master-0/wal/instance:
uuid: "a5b935068d21402a821b68077420d1a9"
format_stamp: "Formatted at 2026-09-23 20:14:16 on dist-test-slave-th4g"
server_key: "1ef29069ca80bd721ec1ea25f608d35f"
server_key_iv: "1d4e3898150fcf5b1f0bd427dd889cdd"
server_key_version: "encryptionkey@0"
I20260923 20:14:16.295011 22363 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.008s sys 0.000s
I20260923 20:14:16.299409 22378 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:14:16.301043 22363 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.002s
I20260923 20:14:16.301255 22363 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790194121577027-1706-0/minicluster-data/master-0/data,/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790194121577027-1706-0/minicluster-data/master-0/wal
uuid: "a5b935068d21402a821b68077420d1a9"
format_stamp: "Formatted at 2026-09-23 20:14:16 on dist-test-slave-th4g"
server_key: "1ef29069ca80bd721ec1ea25f608d35f"
server_key_iv: "1d4e3898150fcf5b1f0bd427dd889cdd"
server_key_version: "encryptionkey@0"
I20260923 20:14:16.301501 22363 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790194121577027-1706-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790194121577027-1706-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790194121577027-1706-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:14:16.308233 22363 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:14:16.312860 22363 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:14:16.313190 22363 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:14:16.336601 22363 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42325
I20260923 20:14:16.336650 22429 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42325 every 8 connection(s)
I20260923 20:14:16.338130 22363 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790194121577027-1706-0/minicluster-data/master-0/data/info.pb
I20260923 20:14:16.343183 22430 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:14:16.347618 1706 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqUuku6/build/asan/bin/kudu as pid 22363
I20260923 20:14:16.347823 1706 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790194121577027-1706-0/minicluster-data/master-0/wal/instance
I20260923 20:14:16.348342 1706 external_mini_cluster.cc:1428] Setting key 34d8ba43e0aa975834ebc00fdc22f975
I20260923 20:14:16.356998 22433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:14:16.350447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59833 (local address 127.0.0.1:42325)
0923 20:14:16.351393 (+ 946us) server_negotiation.cc:206] Beginning negotiation
0923 20:14:16.351404 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0923 20:14:16.351446 (+ 42us) server_negotiation.cc:407] Connection header received
0923 20:14:16.353061 (+ 1615us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:14:16.353085 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:14:16.353474 (+ 389us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:14:16.354125 (+ 651us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:14:16.355317 (+ 1192us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:14:16.355324 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:14:16.355340 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 20:14:16.355358 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:14:16.355430 (+ 72us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 20:14:16.355453 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:14:16.355661 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0923 20:14:16.355776 (+ 115us) server_negotiation.cc:299] Negotiation successful
0923 20:14:16.356022 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":739,"thread_start_us":520,"threads_started":1}
I20260923 20:14:16.357638 22430 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a5b935068d21402a821b68077420d1a9: Bootstrap starting.
I20260923 20:14:16.361461 1706 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 20:14:16.362653 22430 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a5b935068d21402a821b68077420d1a9: Neither blocks nor log segments found. Creating new log.
I20260923 20:14:16.364097 22430 log.cc:824] T 00000000000000000000000000000000 P a5b935068d21402a821b68077420d1a9: Log is configured to *not* fsync() on all Append() calls
I20260923 20:14:16.367843 22430 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a5b935068d21402a821b68077420d1a9: No bootstrap required, opened a new log
I20260923 20:14:16.370507 22433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:14:16.367322 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42263 (local address 127.0.0.1:42325)
0923 20:14:16.367537 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0923 20:14:16.367543 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:14:16.367859 (+ 316us) server_negotiation.cc:407] Connection header received
0923 20:14:16.368305 (+ 446us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:14:16.368312 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:14:16.368424 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:14:16.368590 (+ 166us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:14:16.369685 (+ 1095us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:14:16.369690 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:14:16.369694 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 20:14:16.369717 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:14:16.369775 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 20:14:16.369794 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:14:16.370023 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0923 20:14:16.370090 (+ 67us) server_negotiation.cc:299] Negotiation successful
0923 20:14:16.370146 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260923 20:14:16.378129 22430 raft_consensus.cc:348] T 00000000000000000000000000000000 P a5b935068d21402a821b68077420d1a9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5b935068d21402a821b68077420d1a9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42325 } }
I20260923 20:14:16.378867 22430 raft_consensus.cc:374] T 00000000000000000000000000000000 P a5b935068d21402a821b68077420d1a9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:14:16.379226 22430 raft_consensus.cc:729] T 00000000000000000000000000000000 P a5b935068d21402a821b68077420d1a9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5b935068d21402a821b68077420d1a9, State: Initialized, Role: FOLLOWER
I20260923 20:14:16.380185 22430 consensus_queue.cc:260] T 00000000000000000000000000000000 P a5b935068d21402a821b68077420d1a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5b935068d21402a821b68077420d1a9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42325 } }
I20260923 20:14:16.380496 22430 raft_consensus.cc:388] T 00000000000000000000000000000000 P a5b935068d21402a821b68077420d1a9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 20:14:16.380636 22430 raft_consensus.cc:482] T 00000000000000000000000000000000 P a5b935068d21402a821b68077420d1a9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 20:14:16.380841 22430 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a5b935068d21402a821b68077420d1a9 [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:14:16.384236 22430 raft_consensus.cc:504] T 00000000000000000000000000000000 P a5b935068d21402a821b68077420d1a9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5b935068d21402a821b68077420d1a9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42325 } }
I20260923 20:14:16.384984 22430 leader_election.cc:302] T 00000000000000000000000000000000 P a5b935068d21402a821b68077420d1a9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a5b935068d21402a821b68077420d1a9; no voters:
I20260923 20:14:16.385783 22430 leader_election.cc:288] T 00000000000000000000000000000000 P a5b935068d21402a821b68077420d1a9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 20:14:16.386003 22440 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a5b935068d21402a821b68077420d1a9 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 20:14:16.386834 22440 raft_consensus.cc:686] T 00000000000000000000000000000000 P a5b935068d21402a821b68077420d1a9 [term 1 LEADER]: Becoming Leader. State: Replica: a5b935068d21402a821b68077420d1a9, State: Running, Role: LEADER
I20260923 20:14:16.387411 22440 consensus_queue.cc:237] T 00000000000000000000000000000000 P a5b935068d21402a821b68077420d1a9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5b935068d21402a821b68077420d1a9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42325 } }
I20260923 20:14:16.388113 22430 sys_catalog.cc:564] T 00000000000000000000000000000000 P a5b935068d21402a821b68077420d1a9 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 20:14:16.390821 22442 sys_catalog.cc:455] T 00000000000000000000000000000000 P a5b935068d21402a821b68077420d1a9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a5b935068d21402a821b68077420d1a9. Latest consensus state: current_term: 1 leader_uuid: "a5b935068d21402a821b68077420d1a9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5b935068d21402a821b68077420d1a9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42325 } } }
I20260923 20:14:16.390759 22441 sys_catalog.cc:455] T 00000000000000000000000000000000 P a5b935068d21402a821b68077420d1a9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a5b935068d21402a821b68077420d1a9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5b935068d21402a821b68077420d1a9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42325 } } }
I20260923 20:14:16.391448 22442 sys_catalog.cc:458] T 00000000000000000000000000000000 P a5b935068d21402a821b68077420d1a9 [sys.catalog]: This master's current role is: LEADER
I20260923 20:14:16.391479 22441 sys_catalog.cc:458] T 00000000000000000000000000000000 P a5b935068d21402a821b68077420d1a9 [sys.catalog]: This master's current role is: LEADER
I20260923 20:14:16.392489 22449 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 20:14:16.399892 22449 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 20:14:16.410430 22449 catalog_manager.cc:1347] Generated new cluster ID: ea0448a1230341ebbbf5ececd68dff13
I20260923 20:14:16.410573 22449 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 20:14:16.421376 22449 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 20:14:16.421855 22449 catalog_manager.cc:1504] Loading token signing keys...
I20260923 20:14:16.429036 22449 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a5b935068d21402a821b68077420d1a9: Generated new TSK 0
I20260923 20:14:16.430047 22449 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 20:14:16.437992 1706 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskqUuku6/build/asan/bin/kudu pid 22363
W20260923 20:14:16.439286 22425 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:59833
W20260923 20:14:16.440368 1706 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskqUuku6/build/asan/bin/kudu pid 22363: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260923 20:14:16.440687 1706 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqUuku6/build/asan/bin/kudu with pid 22363
2026-09-23T20:14:16Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1974 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790194121577027-1706-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 20:14:16 dist-test-slave-th4g krb5kdc[22461](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 20:14:16 dist-test-slave-th4g krb5kdc[22461](info): set up 2 sockets
Sep 23 20:14:16 dist-test-slave-th4g krb5kdc[22461](info): commencing operation
krb5kdc: starting...
W20260923 20:14:17.619077 1706 mini_kdc.cc:121] Time spent starting KDC: real 1.139s user 0.000s sys 0.033s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790194121577027-1706-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790194121577027-1706-0/krb5kdc/test-user.keytab.
Sep 23 20:14:17 dist-test-slave-th4g krb5kdc[22461](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194457, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T20:14:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T20:14:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790194121577027-1706-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790194121577027-1706-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260923 20:14:17.902931 1706 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqUuku6/build/asan/bin/kudu
/tmp/dist-test-taskqUuku6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790194121577027-1706-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790194121577027-1706-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790194121577027-1706-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790194121577027-1706-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43761
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790194121577027-1706-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45957
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43761
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790194121577027-1706-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790194121577027-1706-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790194121577027-1706-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:14:18.265318 22474 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:14:18.265686 22474 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:14:18.265784 22474 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:14:18.272843 22474 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 20:14:18.272950 22474 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:14:18.272996 22474 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:14:18.273048 22474 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 20:14:18.273092 22474 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 20:14:18.282948 22474 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45957
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790194121577027-1706-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790194121577027-1706-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43761
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790194121577027-1706-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43761
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790194121577027-1706-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790194121577027-1706-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:14:18.284181 22474 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:14:18.285513 22474 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:14:18.297766 22479 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:14:18.297791 22480 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:14:18.297720 22482 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:14:18.298492 22474 server_base.cc:1034] running on GCE node
I20260923 20:14:18.299247 22474 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:14:18.300860 22474 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:14:18.302069 22474 hybrid_clock.cc:648] HybridClock initialized: now 1790194458302017 us; error 47 us; skew 500 ppm
I20260923 20:14:18.302424 22474 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:14:18 dist-test-slave-th4g krb5kdc[22461](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194458, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:14:18.312712 22474 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260923 20:14:18.314364 22474 webserver.cc:460] Webserver started at http://127.0.0.1:42531/ using document root <none> and password file <none>
I20260923 20:14:18.315321 22474 fs_manager.cc:359] Metadata directory not provided
I20260923 20:14:18.315479 22474 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:14:18.315900 22474 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:14:18.319008 22474 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790194121577027-1706-0/minicluster-data/master-0/data/instance:
uuid: "dde2414d88de42b29b95d9ed730dc803"
format_stamp: "Formatted at 2026-09-23 20:14:18 on dist-test-slave-th4g"
server_key: "d0c3e50bc696a5e555e91464a95985c0"
server_key_iv: "1da1ee82b12fa8dd3a31dd3eabd3fef9"
server_key_version: "encryptionkey@0"
I20260923 20:14:18.319775 22474 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790194121577027-1706-0/minicluster-data/master-0/wal/instance:
uuid: "dde2414d88de42b29b95d9ed730dc803"
format_stamp: "Formatted at 2026-09-23 20:14:18 on dist-test-slave-th4g"
server_key: "d0c3e50bc696a5e555e91464a95985c0"
server_key_iv: "1da1ee82b12fa8dd3a31dd3eabd3fef9"
server_key_version: "encryptionkey@0"
I20260923 20:14:18.325245 22474 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260923 20:14:18.329111 22489 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:14:18.330705 22474 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 20:14:18.330961 22474 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790194121577027-1706-0/minicluster-data/master-0/data,/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790194121577027-1706-0/minicluster-data/master-0/wal
uuid: "dde2414d88de42b29b95d9ed730dc803"
format_stamp: "Formatted at 2026-09-23 20:14:18 on dist-test-slave-th4g"
server_key: "d0c3e50bc696a5e555e91464a95985c0"
server_key_iv: "1da1ee82b12fa8dd3a31dd3eabd3fef9"
server_key_version: "encryptionkey@0"
I20260923 20:14:18.331197 22474 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790194121577027-1706-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790194121577027-1706-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790194121577027-1706-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:14:18.355248 22474 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:14:18.359928 22474 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:14:18.360229 22474 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:14:18.379845 22474 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43761
I20260923 20:14:18.379899 22540 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43761 every 8 connection(s)
I20260923 20:14:18.381237 22474 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790194121577027-1706-0/minicluster-data/master-0/data/info.pb
I20260923 20:14:18.385058 22541 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:14:18.387817 1706 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqUuku6/build/asan/bin/kudu as pid 22474
I20260923 20:14:18.388003 1706 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790194121577027-1706-0/minicluster-data/master-0/wal/instance
I20260923 20:14:18.388484 1706 external_mini_cluster.cc:1428] Setting key fae9cf21ecbc8fcf7fc33e4e8373afea
I20260923 20:14:18.398006 22541 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dde2414d88de42b29b95d9ed730dc803: Bootstrap starting.
I20260923 20:14:18.401520 22541 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dde2414d88de42b29b95d9ed730dc803: Neither blocks nor log segments found. Creating new log.
I20260923 20:14:18.402621 22541 log.cc:824] T 00000000000000000000000000000000 P dde2414d88de42b29b95d9ed730dc803: Log is configured to *not* fsync() on all Append() calls
Sep 23 20:14:18 dist-test-slave-th4g krb5kdc[22461](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194457, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260923 20:14:18.406107 22541 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dde2414d88de42b29b95d9ed730dc803: No bootstrap required, opened a new log
I20260923 20:14:18.414703 22544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:14:18.390460 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44159 (local address 127.0.0.1:43761)
0923 20:14:18.391235 (+ 775us) server_negotiation.cc:206] Beginning negotiation
0923 20:14:18.391259 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0923 20:14:18.391305 (+ 46us) server_negotiation.cc:407] Connection header received
0923 20:14:18.393734 (+ 2429us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:14:18.393765 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:14:18.394376 (+ 611us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:14:18.394819 (+ 443us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:14:18.396262 (+ 1443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:14:18.398239 (+ 1977us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:14:18.399571 (+ 1332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:14:18.400603 (+ 1032us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:14:18.400786 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:14:18.405681 (+ 4895us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:14:18.405694 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:14:18.405714 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:14:18.405765 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:14:18.409960 (+ 4195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:14:18.410724 (+ 764us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:14:18.410735 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:14:18.410749 (+ 14us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:14:18.410952 (+ 203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:14:18.411314 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:14:18.411319 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:14:18.411322 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:14:18.412048 (+ 726us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:14:18.412374 (+ 326us) server_negotiation.cc:1035] Waiting for connection context
0923 20:14:18.412761 (+ 387us) server_negotiation.cc:299] Negotiation successful
0923 20:14:18.413220 (+ 459us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":553,"thread_start_us":341,"threads_started":1}
I20260923 20:14:18.417074 22541 raft_consensus.cc:348] T 00000000000000000000000000000000 P dde2414d88de42b29b95d9ed730dc803 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dde2414d88de42b29b95d9ed730dc803" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43761 } }
I20260923 20:14:18.417539 22541 raft_consensus.cc:374] T 00000000000000000000000000000000 P dde2414d88de42b29b95d9ed730dc803 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:14:18.417644 22541 raft_consensus.cc:729] T 00000000000000000000000000000000 P dde2414d88de42b29b95d9ed730dc803 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dde2414d88de42b29b95d9ed730dc803, State: Initialized, Role: FOLLOWER
I20260923 20:14:18.418354 22541 consensus_queue.cc:260] T 00000000000000000000000000000000 P dde2414d88de42b29b95d9ed730dc803 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dde2414d88de42b29b95d9ed730dc803" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43761 } }
I20260923 20:14:18.418596 22541 raft_consensus.cc:388] T 00000000000000000000000000000000 P dde2414d88de42b29b95d9ed730dc803 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 20:14:18.418732 22541 raft_consensus.cc:482] T 00000000000000000000000000000000 P dde2414d88de42b29b95d9ed730dc803 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 20:14:18.418849 1706 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 20:14:18.418926 22541 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dde2414d88de42b29b95d9ed730dc803 [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:14:18.421056 22541 raft_consensus.cc:504] T 00000000000000000000000000000000 P dde2414d88de42b29b95d9ed730dc803 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dde2414d88de42b29b95d9ed730dc803" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43761 } }
I20260923 20:14:18.421671 22541 leader_election.cc:302] T 00000000000000000000000000000000 P dde2414d88de42b29b95d9ed730dc803 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dde2414d88de42b29b95d9ed730dc803; no voters:
I20260923 20:14:18.422360 22541 leader_election.cc:288] T 00000000000000000000000000000000 P dde2414d88de42b29b95d9ed730dc803 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 20:14:18.422484 22550 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dde2414d88de42b29b95d9ed730dc803 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 20:14:18.424291 22541 sys_catalog.cc:564] T 00000000000000000000000000000000 P dde2414d88de42b29b95d9ed730dc803 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 20:14:18.429773 22550 raft_consensus.cc:686] T 00000000000000000000000000000000 P dde2414d88de42b29b95d9ed730dc803 [term 1 LEADER]: Becoming Leader. State: Replica: dde2414d88de42b29b95d9ed730dc803, State: Running, Role: LEADER
I20260923 20:14:18.430379 22550 consensus_queue.cc:237] T 00000000000000000000000000000000 P dde2414d88de42b29b95d9ed730dc803 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dde2414d88de42b29b95d9ed730dc803" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43761 } }
I20260923 20:14:18.435648 22551 sys_catalog.cc:455] T 00000000000000000000000000000000 P dde2414d88de42b29b95d9ed730dc803 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dde2414d88de42b29b95d9ed730dc803" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dde2414d88de42b29b95d9ed730dc803" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43761 } } }
I20260923 20:14:18.435397 22556 sys_catalog.cc:455] T 00000000000000000000000000000000 P dde2414d88de42b29b95d9ed730dc803 [sys.catalog]: SysCatalogTable state changed. Reason: New leader dde2414d88de42b29b95d9ed730dc803. Latest consensus state: current_term: 1 leader_uuid: "dde2414d88de42b29b95d9ed730dc803" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dde2414d88de42b29b95d9ed730dc803" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43761 } } }
I20260923 20:14:18.437417 22551 sys_catalog.cc:458] T 00000000000000000000000000000000 P dde2414d88de42b29b95d9ed730dc803 [sys.catalog]: This master's current role is: LEADER
I20260923 20:14:18.437737 22556 sys_catalog.cc:458] T 00000000000000000000000000000000 P dde2414d88de42b29b95d9ed730dc803 [sys.catalog]: This master's current role is: LEADER
I20260923 20:14:18.449565 22544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:14:18.423481 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38671 (local address 127.0.0.1:43761)
0923 20:14:18.423657 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0923 20:14:18.423663 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:14:18.429552 (+ 5889us) server_negotiation.cc:407] Connection header received
0923 20:14:18.431046 (+ 1494us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:14:18.431052 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:14:18.431140 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:14:18.431261 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:14:18.432911 (+ 1650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:14:18.434349 (+ 1438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:14:18.436366 (+ 2017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:14:18.438003 (+ 1637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:14:18.438220 (+ 217us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:14:18.441297 (+ 3077us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:14:18.441305 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:14:18.441309 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:14:18.441349 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:14:18.445481 (+ 4132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:14:18.446758 (+ 1277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:14:18.446766 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:14:18.446772 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:14:18.446963 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:14:18.447500 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:14:18.447504 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:14:18.447507 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:14:18.447975 (+ 468us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:14:18.449000 (+ 1025us) server_negotiation.cc:1035] Waiting for connection context
0923 20:14:18.449123 (+ 123us) server_negotiation.cc:299] Negotiation successful
0923 20:14:18.449303 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
W20260923 20:14:18.450768 22565 catalog_manager.cc:1558] T 00000000000000000000000000000000 P dde2414d88de42b29b95d9ed730dc803: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260923 20:14:18.450896 22565 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260923 20:14:18.451036 22566 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 20:14:18.452823 22566 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 20:14:18.461287 22566 catalog_manager.cc:1347] Generated new cluster ID: 87c7b7bbdd054151a8d47aef64a5f732
I20260923 20:14:18.461408 22566 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 20:14:18.472568 22566 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 20:14:18.474035 22566 catalog_manager.cc:1504] Loading token signing keys...
I20260923 20:14:18.486863 22566 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dde2414d88de42b29b95d9ed730dc803: Generated new TSK 0
I20260923 20:14:18.487753 22566 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 20:14:18.499565 1706 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskqUuku6/build/asan/bin/kudu pid 22474
I20260923 20:14:18.500720 22536 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:14:18.592759 1706 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqUuku6/build/asan/bin/kudu with pid 22474
2026-09-23T20:14:18Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (2149 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790194121577027-1706-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 20:14:18 dist-test-slave-th4g krb5kdc[22573](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 20:14:18 dist-test-slave-th4g krb5kdc[22573](info): set up 2 sockets
Sep 23 20:14:18 dist-test-slave-th4g krb5kdc[22573](info): commencing operation
krb5kdc: starting...
W20260923 20:14:19.710978 1706 mini_kdc.cc:121] Time spent starting KDC: real 1.082s user 0.005s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790194121577027-1706-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790194121577027-1706-0/krb5kdc/test-user.keytab.
Sep 23 20:14:19 dist-test-slave-th4g krb5kdc[22573](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194459, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T20:14:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T20:14:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790194121577027-1706-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790194121577027-1706-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260923 20:14:20.029501 1706 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqUuku6/build/asan/bin/kudu
/tmp/dist-test-taskqUuku6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790194121577027-1706-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790194121577027-1706-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790194121577027-1706-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790194121577027-1706-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38949
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790194121577027-1706-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35109
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38949
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790194121577027-1706-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790194121577027-1706-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790194121577027-1706-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:14:20.430758 22586 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:14:20.431141 22586 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:14:20.431327 22586 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:14:20.440920 22586 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 20:14:20.441210 22586 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:14:20.441784 22586 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:14:20.442071 22586 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 20:14:20.442164 22586 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 20:14:20.455494 22586 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35109
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790194121577027-1706-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790194121577027-1706-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38949
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790194121577027-1706-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38949
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790194121577027-1706-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790194121577027-1706-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:14:20.459016 22586 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:14:20.460578 22586 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:14:20.477458 22591 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:14:20.478055 22586 server_base.cc:1034] running on GCE node
W20260923 20:14:20.477813 22592 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:14:20.478118 22594 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:14:20.479614 22586 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:14:20.481992 22586 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:14:20.483374 22586 hybrid_clock.cc:648] HybridClock initialized: now 1790194460483313 us; error 58 us; skew 500 ppm
I20260923 20:14:20.483930 22586 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:14:20 dist-test-slave-th4g krb5kdc[22573](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194460, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:14:20.487910 22586 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260923 20:14:20.490103 22586 webserver.cc:460] Webserver started at http://127.0.0.1:40795/ using document root <none> and password file <none>
I20260923 20:14:20.491158 22586 fs_manager.cc:359] Metadata directory not provided
I20260923 20:14:20.491372 22586 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:14:20.491809 22586 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:14:20.495671 22586 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790194121577027-1706-0/minicluster-data/master-0/data/instance:
uuid: "9d53ad0b93de4a449a1c71ff0e602b08"
format_stamp: "Formatted at 2026-09-23 20:14:20 on dist-test-slave-th4g"
server_key: "9a2bb07fa253537180221e504fbd251a"
server_key_iv: "8719abc3d6abfcc9e7e3bb811ac62748"
server_key_version: "encryptionkey@0"
I20260923 20:14:20.496626 22586 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790194121577027-1706-0/minicluster-data/master-0/wal/instance:
uuid: "9d53ad0b93de4a449a1c71ff0e602b08"
format_stamp: "Formatted at 2026-09-23 20:14:20 on dist-test-slave-th4g"
server_key: "9a2bb07fa253537180221e504fbd251a"
server_key_iv: "8719abc3d6abfcc9e7e3bb811ac62748"
server_key_version: "encryptionkey@0"
I20260923 20:14:20.504962 22586 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.009s sys 0.001s
I20260923 20:14:20.510824 22601 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:14:20.513754 22586 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.002s sys 0.001s
I20260923 20:14:20.514042 22586 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790194121577027-1706-0/minicluster-data/master-0/data,/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790194121577027-1706-0/minicluster-data/master-0/wal
uuid: "9d53ad0b93de4a449a1c71ff0e602b08"
format_stamp: "Formatted at 2026-09-23 20:14:20 on dist-test-slave-th4g"
server_key: "9a2bb07fa253537180221e504fbd251a"
server_key_iv: "8719abc3d6abfcc9e7e3bb811ac62748"
server_key_version: "encryptionkey@0"
I20260923 20:14:20.514605 22586 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790194121577027-1706-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790194121577027-1706-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790194121577027-1706-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:14:20.538926 22586 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:14:20.544526 22586 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:14:20.544879 22586 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:14:20.572909 22586 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38949
I20260923 20:14:20.572937 22652 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38949 every 8 connection(s)
I20260923 20:14:20.575047 22586 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790194121577027-1706-0/minicluster-data/master-0/data/info.pb
I20260923 20:14:20.579736 22653 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:14:20.580531 1706 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqUuku6/build/asan/bin/kudu as pid 22586
I20260923 20:14:20.580755 1706 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790194121577027-1706-0/minicluster-data/master-0/wal/instance
I20260923 20:14:20.581267 1706 external_mini_cluster.cc:1428] Setting key b0019a558879795baa08347a65970f30
I20260923 20:14:20.602945 22656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:14:20.583605 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49873 (local address 127.0.0.1:38949)
0923 20:14:20.585196 (+ 1591us) server_negotiation.cc:206] Beginning negotiation
0923 20:14:20.585516 (+ 320us) server_negotiation.cc:399] Waiting for connection header
0923 20:14:20.585586 (+ 70us) server_negotiation.cc:407] Connection header received
0923 20:14:20.588605 (+ 3019us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:14:20.588637 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:14:20.589159 (+ 522us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:14:20.590135 (+ 976us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:14:20.590810 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:14:20.592590 (+ 1780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:14:20.594498 (+ 1908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:14:20.595242 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:14:20.595464 (+ 222us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:14:20.597300 (+ 1836us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:14:20.597314 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:14:20.597332 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 20:14:20.599235 (+ 1903us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:14:20.599487 (+ 252us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 20:14:20.599566 (+ 79us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:14:20.600436 (+ 870us) server_negotiation.cc:1035] Waiting for connection context
0923 20:14:20.600745 (+ 309us) server_negotiation.cc:299] Negotiation successful
0923 20:14:20.601135 (+ 390us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1344,"thread_start_us":808,"threads_started":1}
I20260923 20:14:20.603278 22653 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9d53ad0b93de4a449a1c71ff0e602b08: Bootstrap starting.
I20260923 20:14:20.611586 22653 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9d53ad0b93de4a449a1c71ff0e602b08: Neither blocks nor log segments found. Creating new log.
I20260923 20:14:20.612136 1706 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 20:14:20.614292 22653 log.cc:824] T 00000000000000000000000000000000 P 9d53ad0b93de4a449a1c71ff0e602b08: Log is configured to *not* fsync() on all Append() calls
I20260923 20:14:20.618723 22653 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9d53ad0b93de4a449a1c71ff0e602b08: No bootstrap required, opened a new log
I20260923 20:14:20.631696 22656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:14:20.620510 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55473 (local address 127.0.0.1:38949)
0923 20:14:20.620771 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0923 20:14:20.620781 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0923 20:14:20.621651 (+ 870us) server_negotiation.cc:407] Connection header received
0923 20:14:20.622077 (+ 426us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:14:20.622081 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:14:20.622164 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:14:20.622270 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:14:20.623528 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:14:20.624542 (+ 1014us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:14:20.627844 (+ 3302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:14:20.628827 (+ 983us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:14:20.628953 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:14:20.630534 (+ 1581us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:14:20.630539 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:14:20.630542 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 20:14:20.630576 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:14:20.630624 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 20:14:20.630641 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:14:20.630739 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0923 20:14:20.630978 (+ 239us) server_negotiation.cc:299] Negotiation successful
0923 20:14:20.631070 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260923 20:14:20.635207 22653 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9d53ad0b93de4a449a1c71ff0e602b08 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d53ad0b93de4a449a1c71ff0e602b08" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38949 } }
I20260923 20:14:20.635680 22653 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9d53ad0b93de4a449a1c71ff0e602b08 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:14:20.635804 22653 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9d53ad0b93de4a449a1c71ff0e602b08 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d53ad0b93de4a449a1c71ff0e602b08, State: Initialized, Role: FOLLOWER
I20260923 20:14:20.636405 22653 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9d53ad0b93de4a449a1c71ff0e602b08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d53ad0b93de4a449a1c71ff0e602b08" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38949 } }
I20260923 20:14:20.636669 22653 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9d53ad0b93de4a449a1c71ff0e602b08 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 20:14:20.636830 22653 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9d53ad0b93de4a449a1c71ff0e602b08 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 20:14:20.637014 22653 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9d53ad0b93de4a449a1c71ff0e602b08 [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:14:20.639786 22653 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9d53ad0b93de4a449a1c71ff0e602b08 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d53ad0b93de4a449a1c71ff0e602b08" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38949 } }
I20260923 20:14:20.640431 22653 leader_election.cc:302] T 00000000000000000000000000000000 P 9d53ad0b93de4a449a1c71ff0e602b08 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9d53ad0b93de4a449a1c71ff0e602b08; no voters:
I20260923 20:14:20.641594 22653 leader_election.cc:288] T 00000000000000000000000000000000 P 9d53ad0b93de4a449a1c71ff0e602b08 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 20:14:20.641846 22663 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9d53ad0b93de4a449a1c71ff0e602b08 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 20:14:20.642668 22663 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9d53ad0b93de4a449a1c71ff0e602b08 [term 1 LEADER]: Becoming Leader. State: Replica: 9d53ad0b93de4a449a1c71ff0e602b08, State: Running, Role: LEADER
I20260923 20:14:20.652082 22663 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9d53ad0b93de4a449a1c71ff0e602b08 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d53ad0b93de4a449a1c71ff0e602b08" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38949 } }
I20260923 20:14:20.652740 22653 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9d53ad0b93de4a449a1c71ff0e602b08 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 20:14:20.659477 22664 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9d53ad0b93de4a449a1c71ff0e602b08 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9d53ad0b93de4a449a1c71ff0e602b08" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d53ad0b93de4a449a1c71ff0e602b08" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38949 } } }
I20260923 20:14:20.660025 22665 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9d53ad0b93de4a449a1c71ff0e602b08 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9d53ad0b93de4a449a1c71ff0e602b08. Latest consensus state: current_term: 1 leader_uuid: "9d53ad0b93de4a449a1c71ff0e602b08" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d53ad0b93de4a449a1c71ff0e602b08" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38949 } } }
I20260923 20:14:20.661075 22664 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9d53ad0b93de4a449a1c71ff0e602b08 [sys.catalog]: This master's current role is: LEADER
I20260923 20:14:20.661075 22665 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9d53ad0b93de4a449a1c71ff0e602b08 [sys.catalog]: This master's current role is: LEADER
I20260923 20:14:20.663504 22672 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 20:14:20.674144 22672 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 20:14:20.687112 22672 catalog_manager.cc:1347] Generated new cluster ID: b5ed0455f8f94e1c8a221444f8399666
I20260923 20:14:20.687469 22672 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 20:14:20.706571 22672 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 20:14:20.708623 22672 catalog_manager.cc:1504] Loading token signing keys...
I20260923 20:14:20.723512 22672 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9d53ad0b93de4a449a1c71ff0e602b08: Generated new TSK 0
I20260923 20:14:20.724798 22672 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 20:14:20.743527 1706 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskqUuku6/build/asan/bin/kudu pid 22586
W20260923 20:14:20.744812 22648 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:49873
W20260923 20:14:20.746399 1706 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskqUuku6/build/asan/bin/kudu pid 22586: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260923 20:14:20.746616 1706 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqUuku6/build/asan/bin/kudu with pid 22586
2026-09-23T20:14:20Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (2154 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790194121577027-1706-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 20:14:20 dist-test-slave-th4g krb5kdc[22684](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 20:14:20 dist-test-slave-th4g krb5kdc[22684](info): set up 2 sockets
Sep 23 20:14:20 dist-test-slave-th4g krb5kdc[22684](info): commencing operation
krb5kdc: starting...
W20260923 20:14:22.027134 1706 mini_kdc.cc:121] Time spent starting KDC: real 1.243s user 0.000s sys 0.032s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790194121577027-1706-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790194121577027-1706-0/krb5kdc/test-user.keytab.
Sep 23 20:14:22 dist-test-slave-th4g krb5kdc[22684](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194462, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T20:14:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T20:14:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790194121577027-1706-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790194121577027-1706-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260923 20:14:22.397272 1706 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqUuku6/build/asan/bin/kudu
/tmp/dist-test-taskqUuku6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790194121577027-1706-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790194121577027-1706-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790194121577027-1706-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790194121577027-1706-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38513
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790194121577027-1706-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36883
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38513
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790194121577027-1706-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790194121577027-1706-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790194121577027-1706-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:14:22.797555 22697 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:14:22.798235 22697 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:14:22.798470 22697 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:14:22.809224 22697 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 20:14:22.809373 22697 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:14:22.809510 22697 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:14:22.809598 22697 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 20:14:22.809669 22697 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 20:14:22.822338 22697 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36883
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790194121577027-1706-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790194121577027-1706-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38513
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790194121577027-1706-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38513
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790194121577027-1706-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790194121577027-1706-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:14:22.823711 22697 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:14:22.825083 22697 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:14:22.838164 22703 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:14:22.838200 22705 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:14:22.838972 22702 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:14:22.839819 22697 server_base.cc:1034] running on GCE node
I20260923 20:14:22.840672 22697 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:14:22.842396 22697 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:14:22.843612 22697 hybrid_clock.cc:648] HybridClock initialized: now 1790194462843547 us; error 55 us; skew 500 ppm
I20260923 20:14:22.844020 22697 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:14:22 dist-test-slave-th4g krb5kdc[22684](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194462, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:14:22.848551 22697 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260923 20:14:22.850486 22697 webserver.cc:460] Webserver started at http://127.0.0.1:36201/ using document root <none> and password file <none>
I20260923 20:14:22.851276 22697 fs_manager.cc:359] Metadata directory not provided
I20260923 20:14:22.851435 22697 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:14:22.851763 22697 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:14:22.855388 22697 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790194121577027-1706-0/minicluster-data/master-0/data/instance:
uuid: "b846dd5a37d34cdda1474e4e496fa350"
format_stamp: "Formatted at 2026-09-23 20:14:22 on dist-test-slave-th4g"
server_key: "68b22a4fb0df922c59d7f7dca3295bee"
server_key_iv: "1d1a9ae74d50e1d722c8b58dcd59f558"
server_key_version: "encryptionkey@0"
I20260923 20:14:22.856297 22697 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790194121577027-1706-0/minicluster-data/master-0/wal/instance:
uuid: "b846dd5a37d34cdda1474e4e496fa350"
format_stamp: "Formatted at 2026-09-23 20:14:22 on dist-test-slave-th4g"
server_key: "68b22a4fb0df922c59d7f7dca3295bee"
server_key_iv: "1d1a9ae74d50e1d722c8b58dcd59f558"
server_key_version: "encryptionkey@0"
I20260923 20:14:22.862095 22697 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260923 20:14:22.866346 22712 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:14:22.867906 22697 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260923 20:14:22.868139 22697 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790194121577027-1706-0/minicluster-data/master-0/data,/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790194121577027-1706-0/minicluster-data/master-0/wal
uuid: "b846dd5a37d34cdda1474e4e496fa350"
format_stamp: "Formatted at 2026-09-23 20:14:22 on dist-test-slave-th4g"
server_key: "68b22a4fb0df922c59d7f7dca3295bee"
server_key_iv: "1d1a9ae74d50e1d722c8b58dcd59f558"
server_key_version: "encryptionkey@0"
I20260923 20:14:22.868381 22697 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790194121577027-1706-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790194121577027-1706-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790194121577027-1706-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:14:22.875190 22697 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:14:22.881393 22697 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:14:22.881804 22697 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:14:22.906760 22697 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38513
I20260923 20:14:22.906814 22763 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38513 every 8 connection(s)
I20260923 20:14:22.908552 22697 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790194121577027-1706-0/minicluster-data/master-0/data/info.pb
I20260923 20:14:22.911036 1706 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqUuku6/build/asan/bin/kudu as pid 22697
I20260923 20:14:22.911288 1706 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790194121577027-1706-0/minicluster-data/master-0/wal/instance
I20260923 20:14:22.911828 1706 external_mini_cluster.cc:1428] Setting key 429800659af5b80673fdddf6890371c4
I20260923 20:14:22.914637 22764 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:14:22.921554 22767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:14:22.914296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34867 (local address 127.0.0.1:38513)
0923 20:14:22.915325 (+ 1029us) server_negotiation.cc:206] Beginning negotiation
0923 20:14:22.915337 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0923 20:14:22.915379 (+ 42us) server_negotiation.cc:407] Connection header received
0923 20:14:22.917423 (+ 2044us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:14:22.917446 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:14:22.917841 (+ 395us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:14:22.918496 (+ 655us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:14:22.919706 (+ 1210us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:14:22.919716 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:14:22.919729 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 20:14:22.919749 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:14:22.919838 (+ 89us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 20:14:22.919861 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:14:22.920049 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0923 20:14:22.920246 (+ 197us) server_negotiation.cc:299] Negotiation successful
0923 20:14:22.920497 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":746,"thread_start_us":481,"threads_started":1}
I20260923 20:14:22.925702 1706 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 20:14:22.929112 22764 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b846dd5a37d34cdda1474e4e496fa350: Bootstrap starting.
I20260923 20:14:22.933357 22764 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b846dd5a37d34cdda1474e4e496fa350: Neither blocks nor log segments found. Creating new log.
I20260923 20:14:22.933652 22767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:14:22.930503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37743 (local address 127.0.0.1:38513)
0923 20:14:22.930716 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0923 20:14:22.930722 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:14:22.931081 (+ 359us) server_negotiation.cc:407] Connection header received
0923 20:14:22.931229 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:14:22.931238 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:14:22.931304 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:14:22.931450 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:14:22.932717 (+ 1267us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:14:22.932722 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:14:22.932727 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 20:14:22.932748 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:14:22.932797 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 20:14:22.932817 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:14:22.932953 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0923 20:14:22.933191 (+ 238us) server_negotiation.cc:299] Negotiation successful
0923 20:14:22.933266 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260923 20:14:22.934805 22764 log.cc:824] T 00000000000000000000000000000000 P b846dd5a37d34cdda1474e4e496fa350: Log is configured to *not* fsync() on all Append() calls
I20260923 20:14:22.938555 22764 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b846dd5a37d34cdda1474e4e496fa350: No bootstrap required, opened a new log
I20260923 20:14:22.949635 22764 raft_consensus.cc:348] T 00000000000000000000000000000000 P b846dd5a37d34cdda1474e4e496fa350 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b846dd5a37d34cdda1474e4e496fa350" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38513 } }
I20260923 20:14:22.950222 22764 raft_consensus.cc:374] T 00000000000000000000000000000000 P b846dd5a37d34cdda1474e4e496fa350 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:14:22.950363 22764 raft_consensus.cc:729] T 00000000000000000000000000000000 P b846dd5a37d34cdda1474e4e496fa350 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b846dd5a37d34cdda1474e4e496fa350, State: Initialized, Role: FOLLOWER
I20260923 20:14:22.951102 22764 consensus_queue.cc:260] T 00000000000000000000000000000000 P b846dd5a37d34cdda1474e4e496fa350 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b846dd5a37d34cdda1474e4e496fa350" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38513 } }
I20260923 20:14:22.951339 22764 raft_consensus.cc:388] T 00000000000000000000000000000000 P b846dd5a37d34cdda1474e4e496fa350 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 20:14:22.951483 22764 raft_consensus.cc:482] T 00000000000000000000000000000000 P b846dd5a37d34cdda1474e4e496fa350 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 20:14:22.951658 22764 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b846dd5a37d34cdda1474e4e496fa350 [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:14:22.954501 22764 raft_consensus.cc:504] T 00000000000000000000000000000000 P b846dd5a37d34cdda1474e4e496fa350 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b846dd5a37d34cdda1474e4e496fa350" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38513 } }
I20260923 20:14:22.955082 22764 leader_election.cc:302] T 00000000000000000000000000000000 P b846dd5a37d34cdda1474e4e496fa350 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b846dd5a37d34cdda1474e4e496fa350; no voters:
I20260923 20:14:22.956071 22764 leader_election.cc:288] T 00000000000000000000000000000000 P b846dd5a37d34cdda1474e4e496fa350 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 20:14:22.956231 22774 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b846dd5a37d34cdda1474e4e496fa350 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 20:14:22.957047 22774 raft_consensus.cc:686] T 00000000000000000000000000000000 P b846dd5a37d34cdda1474e4e496fa350 [term 1 LEADER]: Becoming Leader. State: Replica: b846dd5a37d34cdda1474e4e496fa350, State: Running, Role: LEADER
I20260923 20:14:22.957840 22774 consensus_queue.cc:237] T 00000000000000000000000000000000 P b846dd5a37d34cdda1474e4e496fa350 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b846dd5a37d34cdda1474e4e496fa350" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38513 } }
I20260923 20:14:22.958577 22764 sys_catalog.cc:564] T 00000000000000000000000000000000 P b846dd5a37d34cdda1474e4e496fa350 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 20:14:22.961628 22776 sys_catalog.cc:455] T 00000000000000000000000000000000 P b846dd5a37d34cdda1474e4e496fa350 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b846dd5a37d34cdda1474e4e496fa350. Latest consensus state: current_term: 1 leader_uuid: "b846dd5a37d34cdda1474e4e496fa350" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b846dd5a37d34cdda1474e4e496fa350" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38513 } } }
I20260923 20:14:22.962514 22776 sys_catalog.cc:458] T 00000000000000000000000000000000 P b846dd5a37d34cdda1474e4e496fa350 [sys.catalog]: This master's current role is: LEADER
I20260923 20:14:22.961388 22775 sys_catalog.cc:455] T 00000000000000000000000000000000 P b846dd5a37d34cdda1474e4e496fa350 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b846dd5a37d34cdda1474e4e496fa350" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b846dd5a37d34cdda1474e4e496fa350" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38513 } } }
I20260923 20:14:22.965161 22775 sys_catalog.cc:458] T 00000000000000000000000000000000 P b846dd5a37d34cdda1474e4e496fa350 [sys.catalog]: This master's current role is: LEADER
I20260923 20:14:22.965173 22781 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 20:14:22.972985 22781 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 20:14:22.986193 22781 catalog_manager.cc:1347] Generated new cluster ID: 40a4fad1d9674389bb58e3e8aafeea0f
I20260923 20:14:22.986377 22781 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 20:14:23.009378 22781 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 20:14:23.009994 22781 catalog_manager.cc:1504] Loading token signing keys...
I20260923 20:14:23.021402 22781 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b846dd5a37d34cdda1474e4e496fa350: Generated new TSK 0
I20260923 20:14:23.022423 22781 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 20:14:23.040310 1706 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskqUuku6/build/asan/bin/kudu pid 22697
W20260923 20:14:23.041455 22759 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:34867
W20260923 20:14:23.042464 1706 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskqUuku6/build/asan/bin/kudu pid 22697: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260923 20:14:23.042779 1706 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqUuku6/build/asan/bin/kudu with pid 22697
2026-09-23T20:14:23Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (2299 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790194121577027-1706-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 20:14:23 dist-test-slave-th4g krb5kdc[22795](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 20:14:23 dist-test-slave-th4g krb5kdc[22795](info): set up 2 sockets
Sep 23 20:14:23 dist-test-slave-th4g krb5kdc[22795](info): commencing operation
krb5kdc: starting...
W20260923 20:14:24.174669 1706 mini_kdc.cc:121] Time spent starting KDC: real 1.090s user 0.000s sys 0.032s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790194121577027-1706-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790194121577027-1706-0/krb5kdc/test-user.keytab.
Sep 23 20:14:24 dist-test-slave-th4g krb5kdc[22795](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194464, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T20:14:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T20:14:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790194121577027-1706-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790194121577027-1706-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260923 20:14:24.452316 1706 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqUuku6/build/asan/bin/kudu
/tmp/dist-test-taskqUuku6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790194121577027-1706-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790194121577027-1706-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790194121577027-1706-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790194121577027-1706-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35321
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790194121577027-1706-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46043
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35321
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790194121577027-1706-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790194121577027-1706-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790194121577027-1706-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:14:24.818427 22808 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:14:24.818733 22808 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:14:24.818914 22808 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:14:24.827800 22808 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 20:14:24.827940 22808 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:14:24.828063 22808 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:14:24.828199 22808 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 20:14:24.828272 22808 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 20:14:24.839748 22808 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46043
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790194121577027-1706-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790194121577027-1706-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35321
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790194121577027-1706-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35321
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790194121577027-1706-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790194121577027-1706-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:14:24.841164 22808 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:14:24.842566 22808 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:14:24.855784 22813 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:14:24.855832 22814 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:14:24.856493 22816 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:14:24.856899 22808 server_base.cc:1034] running on GCE node
I20260923 20:14:24.857730 22808 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:14:24.859272 22808 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:14:24.860507 22808 hybrid_clock.cc:648] HybridClock initialized: now 1790194464860432 us; error 40 us; skew 500 ppm
I20260923 20:14:24.860881 22808 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:14:24 dist-test-slave-th4g krb5kdc[22795](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194464, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:14:24.871785 22808 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260923 20:14:24.873638 22808 webserver.cc:460] Webserver started at http://127.0.0.1:43971/ using document root <none> and password file <none>
I20260923 20:14:24.874521 22808 fs_manager.cc:359] Metadata directory not provided
I20260923 20:14:24.874693 22808 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:14:24.875041 22808 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:14:24.878495 22808 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790194121577027-1706-0/minicluster-data/master-0/data/instance:
uuid: "bccfe75db2634a38a414039bf03e495e"
format_stamp: "Formatted at 2026-09-23 20:14:24 on dist-test-slave-th4g"
server_key: "6a50dc8c3d36ea03074d1a90c2d00582"
server_key_iv: "f30ad7e98a188368f0122aacb7ac5b81"
server_key_version: "encryptionkey@0"
I20260923 20:14:24.879385 22808 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790194121577027-1706-0/minicluster-data/master-0/wal/instance:
uuid: "bccfe75db2634a38a414039bf03e495e"
format_stamp: "Formatted at 2026-09-23 20:14:24 on dist-test-slave-th4g"
server_key: "6a50dc8c3d36ea03074d1a90c2d00582"
server_key_iv: "f30ad7e98a188368f0122aacb7ac5b81"
server_key_version: "encryptionkey@0"
I20260923 20:14:24.885375 22808 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260923 20:14:24.889755 22823 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:14:24.891558 22808 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260923 20:14:24.891791 22808 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790194121577027-1706-0/minicluster-data/master-0/data,/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790194121577027-1706-0/minicluster-data/master-0/wal
uuid: "bccfe75db2634a38a414039bf03e495e"
format_stamp: "Formatted at 2026-09-23 20:14:24 on dist-test-slave-th4g"
server_key: "6a50dc8c3d36ea03074d1a90c2d00582"
server_key_iv: "f30ad7e98a188368f0122aacb7ac5b81"
server_key_version: "encryptionkey@0"
I20260923 20:14:24.892035 22808 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790194121577027-1706-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790194121577027-1706-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790194121577027-1706-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:14:24.905786 22808 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:14:24.910257 22808 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:14:24.910607 22808 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:14:24.932891 22808 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35321
I20260923 20:14:24.932991 22874 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35321 every 8 connection(s)
I20260923 20:14:24.934458 22808 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790194121577027-1706-0/minicluster-data/master-0/data/info.pb
I20260923 20:14:24.938513 22875 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:14:24.942556 1706 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqUuku6/build/asan/bin/kudu as pid 22808
I20260923 20:14:24.942765 1706 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790194121577027-1706-0/minicluster-data/master-0/wal/instance
I20260923 20:14:24.943369 1706 external_mini_cluster.cc:1428] Setting key 407af6a6171cc0292d6730bae8fa2fa8
I20260923 20:14:24.953357 22875 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bccfe75db2634a38a414039bf03e495e: Bootstrap starting.
I20260923 20:14:24.957086 22875 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bccfe75db2634a38a414039bf03e495e: Neither blocks nor log segments found. Creating new log.
I20260923 20:14:24.958175 22875 log.cc:824] T 00000000000000000000000000000000 P bccfe75db2634a38a414039bf03e495e: Log is configured to *not* fsync() on all Append() calls
Sep 23 20:14:24 dist-test-slave-th4g krb5kdc[22795](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194464, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260923 20:14:24.961314 22875 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bccfe75db2634a38a414039bf03e495e: No bootstrap required, opened a new log
I20260923 20:14:24.969655 22878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:14:24.945180 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.239:46149 (local address 127.0.0.1:35321)
0923 20:14:24.946025 (+ 845us) server_negotiation.cc:206] Beginning negotiation
0923 20:14:24.946042 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0923 20:14:24.946091 (+ 49us) server_negotiation.cc:407] Connection header received
0923 20:14:24.948470 (+ 2379us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:14:24.948503 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:14:24.949028 (+ 525us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:14:24.949492 (+ 464us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:14:24.951003 (+ 1511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:14:24.952971 (+ 1968us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:14:24.954492 (+ 1521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:14:24.955611 (+ 1119us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:14:24.955803 (+ 192us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:14:24.960399 (+ 4596us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:14:24.960408 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:14:24.960423 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:14:24.960458 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:14:24.964135 (+ 3677us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:14:24.965401 (+ 1266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:14:24.965408 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:14:24.965415 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:14:24.965607 (+ 192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:14:24.966227 (+ 620us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:14:24.966232 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:14:24.966234 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:14:24.967139 (+ 905us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:14:24.967426 (+ 287us) server_negotiation.cc:1035] Waiting for connection context
0923 20:14:24.968014 (+ 588us) server_negotiation.cc:299] Negotiation successful
0923 20:14:24.968496 (+ 482us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":572,"thread_start_us":337,"threads_started":1}
I20260923 20:14:24.970628 22875 raft_consensus.cc:348] T 00000000000000000000000000000000 P bccfe75db2634a38a414039bf03e495e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bccfe75db2634a38a414039bf03e495e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35321 } }
I20260923 20:14:24.971055 22875 raft_consensus.cc:374] T 00000000000000000000000000000000 P bccfe75db2634a38a414039bf03e495e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:14:24.971197 22875 raft_consensus.cc:729] T 00000000000000000000000000000000 P bccfe75db2634a38a414039bf03e495e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bccfe75db2634a38a414039bf03e495e, State: Initialized, Role: FOLLOWER
I20260923 20:14:24.971899 22875 consensus_queue.cc:260] T 00000000000000000000000000000000 P bccfe75db2634a38a414039bf03e495e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bccfe75db2634a38a414039bf03e495e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35321 } }
I20260923 20:14:24.972210 22875 raft_consensus.cc:388] T 00000000000000000000000000000000 P bccfe75db2634a38a414039bf03e495e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 20:14:24.972412 22875 raft_consensus.cc:482] T 00000000000000000000000000000000 P bccfe75db2634a38a414039bf03e495e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 20:14:24.972597 22875 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bccfe75db2634a38a414039bf03e495e [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:14:24.974347 1706 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 20:14:24.975875 22875 raft_consensus.cc:504] T 00000000000000000000000000000000 P bccfe75db2634a38a414039bf03e495e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bccfe75db2634a38a414039bf03e495e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35321 } }
I20260923 20:14:24.976460 22875 leader_election.cc:302] T 00000000000000000000000000000000 P bccfe75db2634a38a414039bf03e495e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bccfe75db2634a38a414039bf03e495e; no voters:
I20260923 20:14:24.977155 22875 leader_election.cc:288] T 00000000000000000000000000000000 P bccfe75db2634a38a414039bf03e495e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 20:14:24.977284 22880 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bccfe75db2634a38a414039bf03e495e [term 1 FOLLOWER]: Leader election won for term 1
I20260923 20:14:24.979110 22880 raft_consensus.cc:686] T 00000000000000000000000000000000 P bccfe75db2634a38a414039bf03e495e [term 1 LEADER]: Becoming Leader. State: Replica: bccfe75db2634a38a414039bf03e495e, State: Running, Role: LEADER
I20260923 20:14:24.979715 22880 consensus_queue.cc:237] T 00000000000000000000000000000000 P bccfe75db2634a38a414039bf03e495e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bccfe75db2634a38a414039bf03e495e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35321 } }
I20260923 20:14:24.979841 22875 sys_catalog.cc:564] T 00000000000000000000000000000000 P bccfe75db2634a38a414039bf03e495e [sys.catalog]: configured and running, proceeding with master startup.
I20260923 20:14:24.982358 22883 sys_catalog.cc:455] T 00000000000000000000000000000000 P bccfe75db2634a38a414039bf03e495e [sys.catalog]: SysCatalogTable state changed. Reason: New leader bccfe75db2634a38a414039bf03e495e. Latest consensus state: current_term: 1 leader_uuid: "bccfe75db2634a38a414039bf03e495e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bccfe75db2634a38a414039bf03e495e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35321 } } }
I20260923 20:14:24.982767 22883 sys_catalog.cc:458] T 00000000000000000000000000000000 P bccfe75db2634a38a414039bf03e495e [sys.catalog]: This master's current role is: LEADER
I20260923 20:14:24.983328 22881 sys_catalog.cc:455] T 00000000000000000000000000000000 P bccfe75db2634a38a414039bf03e495e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bccfe75db2634a38a414039bf03e495e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bccfe75db2634a38a414039bf03e495e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35321 } } }
I20260923 20:14:24.983729 22881 sys_catalog.cc:458] T 00000000000000000000000000000000 P bccfe75db2634a38a414039bf03e495e [sys.catalog]: This master's current role is: LEADER
I20260923 20:14:24.986100 22889 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 20:14:24.991873 22889 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 20:14:25.005659 22878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:14:24.983401 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.239:54703 (local address 127.0.0.1:35321)
0923 20:14:24.983985 (+ 584us) server_negotiation.cc:206] Beginning negotiation
0923 20:14:24.983992 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:14:24.984016 (+ 24us) server_negotiation.cc:407] Connection header received
0923 20:14:24.984213 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:14:24.984222 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:14:24.984332 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:14:24.984421 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:14:24.986935 (+ 2514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:14:24.987796 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:14:24.989915 (+ 2119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:14:24.990900 (+ 985us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:14:24.991047 (+ 147us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:14:24.995465 (+ 4418us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:14:24.995475 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:14:24.995484 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:14:24.995543 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:14:25.000421 (+ 4878us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:14:25.001737 (+ 1316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:14:25.001747 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:14:25.001753 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:14:25.001989 (+ 236us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:14:25.002728 (+ 739us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:14:25.002736 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:14:25.002741 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:14:25.003347 (+ 606us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:14:25.003676 (+ 329us) server_negotiation.cc:1035] Waiting for connection context
0923 20:14:25.004809 (+ 1133us) server_negotiation.cc:299] Negotiation successful
0923 20:14:25.005095 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":465}
I20260923 20:14:25.013935 22889 catalog_manager.cc:1347] Generated new cluster ID: 47af5aa88e97412eb7480467e2bc09fc
I20260923 20:14:25.014060 22889 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 20:14:25.041777 22889 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 20:14:25.043676 22889 catalog_manager.cc:1504] Loading token signing keys...
I20260923 20:14:25.055124 22889 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bccfe75db2634a38a414039bf03e495e: Generated new TSK 0
I20260923 20:14:25.056200 22889 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 20:14:25.072932 1706 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskqUuku6/build/asan/bin/kudu pid 22808
I20260923 20:14:25.074072 22870 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:14:25.167939 1706 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqUuku6/build/asan/bin/kudu with pid 22808
W20260923 20:14:25.185768 22793 connection.cc:535] client connection to 127.0.0.1:35321 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:35321): Connection reset by peer (error 104)
2026-09-23T20:14:25Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (2121 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790194121577027-1706-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 20:14:25 dist-test-slave-th4g krb5kdc[22907](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 20:14:25 dist-test-slave-th4g krb5kdc[22907](info): set up 2 sockets
Sep 23 20:14:25 dist-test-slave-th4g krb5kdc[22907](info): commencing operation
krb5kdc: starting...
W20260923 20:14:26.322904 1706 mini_kdc.cc:121] Time spent starting KDC: real 1.118s user 0.003s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790194121577027-1706-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790194121577027-1706-0/krb5kdc/test-user.keytab.
Sep 23 20:14:26 dist-test-slave-th4g krb5kdc[22907](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194466, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T20:14:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T20:14:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790194121577027-1706-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790194121577027-1706-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260923 20:14:26.622465 1706 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqUuku6/build/asan/bin/kudu
/tmp/dist-test-taskqUuku6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790194121577027-1706-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790194121577027-1706-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790194121577027-1706-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790194121577027-1706-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42699
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790194121577027-1706-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35903
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42699
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790194121577027-1706-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790194121577027-1706-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790194121577027-1706-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:14:26.986547 22920 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:14:26.986886 22920 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:14:26.986994 22920 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:14:26.995007 22920 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 20:14:26.995152 22920 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:14:26.995203 22920 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:14:26.995258 22920 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 20:14:26.995298 22920 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 20:14:27.006191 22920 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35903
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790194121577027-1706-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790194121577027-1706-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42699
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790194121577027-1706-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42699
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790194121577027-1706-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790194121577027-1706-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:14:27.007547 22920 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:14:27.009011 22920 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:14:27.024843 22925 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:14:27.025246 22926 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:14:27.026593 22928 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:14:27.027714 22920 server_base.cc:1034] running on GCE node
I20260923 20:14:27.028741 22920 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:14:27.030563 22920 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:14:27.031845 22920 hybrid_clock.cc:648] HybridClock initialized: now 1790194467031789 us; error 68 us; skew 500 ppm
I20260923 20:14:27.032305 22920 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:14:27 dist-test-slave-th4g krb5kdc[22907](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194467, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:14:27.036616 22920 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260923 20:14:27.038880 22920 webserver.cc:460] Webserver started at http://127.0.0.1:41345/ using document root <none> and password file <none>
I20260923 20:14:27.039953 22920 fs_manager.cc:359] Metadata directory not provided
I20260923 20:14:27.040113 22920 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:14:27.040562 22920 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:14:27.044957 22920 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790194121577027-1706-0/minicluster-data/master-0/data/instance:
uuid: "3fce4859e3a74f7d9db373423e277c0f"
format_stamp: "Formatted at 2026-09-23 20:14:27 on dist-test-slave-th4g"
server_key: "cff909862d95ab6f3639477658087f27"
server_key_iv: "e3c8799ca0c16bff9b209ce66cc4c105"
server_key_version: "encryptionkey@0"
I20260923 20:14:27.046135 22920 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790194121577027-1706-0/minicluster-data/master-0/wal/instance:
uuid: "3fce4859e3a74f7d9db373423e277c0f"
format_stamp: "Formatted at 2026-09-23 20:14:27 on dist-test-slave-th4g"
server_key: "cff909862d95ab6f3639477658087f27"
server_key_iv: "e3c8799ca0c16bff9b209ce66cc4c105"
server_key_version: "encryptionkey@0"
I20260923 20:14:27.054464 22920 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.006s sys 0.001s
I20260923 20:14:27.060153 22935 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:14:27.062155 22920 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260923 20:14:27.062389 22920 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790194121577027-1706-0/minicluster-data/master-0/data,/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790194121577027-1706-0/minicluster-data/master-0/wal
uuid: "3fce4859e3a74f7d9db373423e277c0f"
format_stamp: "Formatted at 2026-09-23 20:14:27 on dist-test-slave-th4g"
server_key: "cff909862d95ab6f3639477658087f27"
server_key_iv: "e3c8799ca0c16bff9b209ce66cc4c105"
server_key_version: "encryptionkey@0"
I20260923 20:14:27.062675 22920 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790194121577027-1706-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790194121577027-1706-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790194121577027-1706-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:14:27.092702 22920 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:14:27.097775 22920 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:14:27.098167 22920 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:14:27.119602 22920 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42699
I20260923 20:14:27.119620 22986 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42699 every 8 connection(s)
I20260923 20:14:27.121074 22920 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790194121577027-1706-0/minicluster-data/master-0/data/info.pb
I20260923 20:14:27.121447 1706 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqUuku6/build/asan/bin/kudu as pid 22920
I20260923 20:14:27.121634 1706 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790194121577027-1706-0/minicluster-data/master-0/wal/instance
I20260923 20:14:27.122284 1706 external_mini_cluster.cc:1428] Setting key e5d323ac07bf81451c136d5c7222550d
I20260923 20:14:27.126086 22987 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:14:27.138778 22990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:14:27.124398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.239:59763 (local address 127.0.0.1:42699)
0923 20:14:27.125195 (+ 797us) server_negotiation.cc:206] Beginning negotiation
0923 20:14:27.125210 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0923 20:14:27.125258 (+ 48us) server_negotiation.cc:407] Connection header received
0923 20:14:27.127700 (+ 2442us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:14:27.127732 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:14:27.128291 (+ 559us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:14:27.128788 (+ 497us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:14:27.129370 (+ 582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:14:27.131397 (+ 2027us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:14:27.133043 (+ 1646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:14:27.134163 (+ 1120us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:14:27.134356 (+ 193us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:14:27.135730 (+ 1374us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:14:27.135741 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:14:27.135763 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 20:14:27.136625 (+ 862us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:14:27.136733 (+ 108us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 20:14:27.136780 (+ 47us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:14:27.136944 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0923 20:14:27.137279 (+ 335us) server_negotiation.cc:299] Negotiation successful
0923 20:14:27.137472 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":566,"thread_start_us":325,"threads_started":1}
I20260923 20:14:27.140345 22987 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3fce4859e3a74f7d9db373423e277c0f: Bootstrap starting.
I20260923 20:14:27.143617 1706 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 20:14:27.144081 22987 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3fce4859e3a74f7d9db373423e277c0f: Neither blocks nor log segments found. Creating new log.
I20260923 20:14:27.145357 22987 log.cc:824] T 00000000000000000000000000000000 P 3fce4859e3a74f7d9db373423e277c0f: Log is configured to *not* fsync() on all Append() calls
I20260923 20:14:27.148301 22987 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3fce4859e3a74f7d9db373423e277c0f: No bootstrap required, opened a new log
I20260923 20:14:27.156333 22990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:14:27.148792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.239:47693 (local address 127.0.0.1:42699)
0923 20:14:27.148986 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0923 20:14:27.148992 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:14:27.149558 (+ 566us) server_negotiation.cc:407] Connection header received
0923 20:14:27.149720 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:14:27.149725 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:14:27.149823 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:14:27.149969 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:14:27.150421 (+ 452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:14:27.151627 (+ 1206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:14:27.153187 (+ 1560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:14:27.154001 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:14:27.154127 (+ 126us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:14:27.155429 (+ 1302us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:14:27.155441 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:14:27.155447 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 20:14:27.155479 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:14:27.155540 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 20:14:27.155564 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:14:27.155712 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0923 20:14:27.155924 (+ 212us) server_negotiation.cc:299] Negotiation successful
0923 20:14:27.155980 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260923 20:14:27.158715 22987 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3fce4859e3a74f7d9db373423e277c0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3fce4859e3a74f7d9db373423e277c0f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42699 } }
I20260923 20:14:27.159085 22987 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3fce4859e3a74f7d9db373423e277c0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:14:27.159233 22987 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3fce4859e3a74f7d9db373423e277c0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3fce4859e3a74f7d9db373423e277c0f, State: Initialized, Role: FOLLOWER
I20260923 20:14:27.159888 22987 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3fce4859e3a74f7d9db373423e277c0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3fce4859e3a74f7d9db373423e277c0f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42699 } }
I20260923 20:14:27.160120 22987 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3fce4859e3a74f7d9db373423e277c0f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 20:14:27.160283 22987 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3fce4859e3a74f7d9db373423e277c0f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 20:14:27.160462 22987 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3fce4859e3a74f7d9db373423e277c0f [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:14:27.163223 22987 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3fce4859e3a74f7d9db373423e277c0f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3fce4859e3a74f7d9db373423e277c0f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42699 } }
I20260923 20:14:27.163780 22987 leader_election.cc:302] T 00000000000000000000000000000000 P 3fce4859e3a74f7d9db373423e277c0f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3fce4859e3a74f7d9db373423e277c0f; no voters:
I20260923 20:14:27.164601 22987 leader_election.cc:288] T 00000000000000000000000000000000 P 3fce4859e3a74f7d9db373423e277c0f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 20:14:27.164729 22997 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3fce4859e3a74f7d9db373423e277c0f [term 1 FOLLOWER]: Leader election won for term 1
I20260923 20:14:27.165581 22997 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3fce4859e3a74f7d9db373423e277c0f [term 1 LEADER]: Becoming Leader. State: Replica: 3fce4859e3a74f7d9db373423e277c0f, State: Running, Role: LEADER
I20260923 20:14:27.166188 22997 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3fce4859e3a74f7d9db373423e277c0f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3fce4859e3a74f7d9db373423e277c0f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42699 } }
I20260923 20:14:27.167032 22987 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3fce4859e3a74f7d9db373423e277c0f [sys.catalog]: configured and running, proceeding with master startup.
I20260923 20:14:27.169312 22999 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3fce4859e3a74f7d9db373423e277c0f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3fce4859e3a74f7d9db373423e277c0f. Latest consensus state: current_term: 1 leader_uuid: "3fce4859e3a74f7d9db373423e277c0f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3fce4859e3a74f7d9db373423e277c0f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42699 } } }
I20260923 20:14:27.169461 22998 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3fce4859e3a74f7d9db373423e277c0f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3fce4859e3a74f7d9db373423e277c0f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3fce4859e3a74f7d9db373423e277c0f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42699 } } }
I20260923 20:14:27.170089 22998 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3fce4859e3a74f7d9db373423e277c0f [sys.catalog]: This master's current role is: LEADER
I20260923 20:14:27.170094 22999 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3fce4859e3a74f7d9db373423e277c0f [sys.catalog]: This master's current role is: LEADER
I20260923 20:14:27.172374 23005 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 20:14:27.177335 23005 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 20:14:27.186333 23005 catalog_manager.cc:1347] Generated new cluster ID: 071029950cfb4575a78d500f66a442b8
I20260923 20:14:27.186476 23005 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 20:14:27.216223 23005 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 20:14:27.218358 23005 catalog_manager.cc:1504] Loading token signing keys...
I20260923 20:14:27.229580 23005 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3fce4859e3a74f7d9db373423e277c0f: Generated new TSK 0
I20260923 20:14:27.230578 23005 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 20:14:27.236889 1706 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskqUuku6/build/asan/bin/kudu pid 22920
W20260923 20:14:27.238309 22982 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.239:59763
W20260923 20:14:27.239475 1706 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskqUuku6/build/asan/bin/kudu pid 22920: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260923 20:14:27.239638 1706 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqUuku6/build/asan/bin/kudu with pid 22920
W20260923 20:14:27.257229 22905 connection.cc:535] client connection to 127.0.0.1:42699 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:42699): Connection reset by peer (error 104)
2026-09-23T20:14:27Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (2071 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790194121577027-1706-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 20:14:27 dist-test-slave-th4g krb5kdc[23018](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 20:14:27 dist-test-slave-th4g krb5kdc[23018](info): set up 2 sockets
Sep 23 20:14:27 dist-test-slave-th4g krb5kdc[23018](info): commencing operation
krb5kdc: starting...
W20260923 20:14:28.386716 1706 mini_kdc.cc:121] Time spent starting KDC: real 1.109s user 0.002s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790194121577027-1706-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790194121577027-1706-0/krb5kdc/test-user.keytab.
Sep 23 20:14:28 dist-test-slave-th4g krb5kdc[23018](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194468, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T20:14:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T20:14:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790194121577027-1706-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790194121577027-1706-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260923 20:14:28.654127 1706 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqUuku6/build/asan/bin/kudu
/tmp/dist-test-taskqUuku6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790194121577027-1706-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790194121577027-1706-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790194121577027-1706-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790194121577027-1706-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37921
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790194121577027-1706-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40091
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37921
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790194121577027-1706-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790194121577027-1706-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790194121577027-1706-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:14:28.993029 23031 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:14:28.993405 23031 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:14:28.993582 23031 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:14:29.001051 23031 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 20:14:29.001232 23031 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:14:29.001343 23031 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:14:29.001449 23031 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 20:14:29.001533 23031 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 20:14:29.011881 23031 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40091
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790194121577027-1706-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790194121577027-1706-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37921
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790194121577027-1706-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37921
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790194121577027-1706-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790194121577027-1706-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:14:29.013307 23031 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:14:29.014693 23031 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:14:29.027238 23036 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:14:29.027165 23037 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:14:29.027207 23039 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:14:29.028007 23031 server_base.cc:1034] running on GCE node
I20260923 20:14:29.028748 23031 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:14:29.030247 23031 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:14:29.031476 23031 hybrid_clock.cc:648] HybridClock initialized: now 1790194469031433 us; error 29 us; skew 500 ppm
I20260923 20:14:29.031836 23031 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:14:29 dist-test-slave-th4g krb5kdc[23018](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194469, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:14:29.035678 23031 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260923 20:14:29.037505 23031 webserver.cc:460] Webserver started at http://127.0.0.1:40847/ using document root <none> and password file <none>
I20260923 20:14:29.038316 23031 fs_manager.cc:359] Metadata directory not provided
I20260923 20:14:29.038475 23031 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:14:29.038803 23031 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:14:29.041805 23031 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790194121577027-1706-0/minicluster-data/master-0/data/instance:
uuid: "2b165209a0ca415f928f259bdc8fb0d5"
format_stamp: "Formatted at 2026-09-23 20:14:29 on dist-test-slave-th4g"
server_key: "d6f45570b825b118b0b3389814cc28ac"
server_key_iv: "b999d2f781e40b06771efbdbf46fd804"
server_key_version: "encryptionkey@0"
I20260923 20:14:29.042654 23031 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790194121577027-1706-0/minicluster-data/master-0/wal/instance:
uuid: "2b165209a0ca415f928f259bdc8fb0d5"
format_stamp: "Formatted at 2026-09-23 20:14:29 on dist-test-slave-th4g"
server_key: "d6f45570b825b118b0b3389814cc28ac"
server_key_iv: "b999d2f781e40b06771efbdbf46fd804"
server_key_version: "encryptionkey@0"
I20260923 20:14:29.048002 23031 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260923 20:14:29.051697 23046 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:14:29.053177 23031 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260923 20:14:29.053390 23031 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790194121577027-1706-0/minicluster-data/master-0/data,/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790194121577027-1706-0/minicluster-data/master-0/wal
uuid: "2b165209a0ca415f928f259bdc8fb0d5"
format_stamp: "Formatted at 2026-09-23 20:14:29 on dist-test-slave-th4g"
server_key: "d6f45570b825b118b0b3389814cc28ac"
server_key_iv: "b999d2f781e40b06771efbdbf46fd804"
server_key_version: "encryptionkey@0"
I20260923 20:14:29.053618 23031 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790194121577027-1706-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790194121577027-1706-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790194121577027-1706-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:14:29.059592 23031 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:14:29.063941 23031 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:14:29.064268 23031 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:14:29.083650 23031 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37921
I20260923 20:14:29.083688 23097 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37921 every 8 connection(s)
I20260923 20:14:29.085068 23031 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790194121577027-1706-0/minicluster-data/master-0/data/info.pb
I20260923 20:14:29.089154 23098 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:14:29.089673 1706 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqUuku6/build/asan/bin/kudu as pid 23031
I20260923 20:14:29.089848 1706 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790194121577027-1706-0/minicluster-data/master-0/wal/instance
I20260923 20:14:29.090348 1706 external_mini_cluster.cc:1428] Setting key fcde7f5a920f9b329a9912b23ee60286
I20260923 20:14:29.099310 23101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:14:29.092419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.239:59571 (local address 127.0.0.1:37921)
0923 20:14:29.093280 (+ 861us) server_negotiation.cc:206] Beginning negotiation
0923 20:14:29.093292 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0923 20:14:29.093333 (+ 41us) server_negotiation.cc:407] Connection header received
0923 20:14:29.095725 (+ 2392us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:14:29.095750 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:14:29.096153 (+ 403us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:14:29.096542 (+ 389us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:14:29.097631 (+ 1089us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:14:29.097638 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:14:29.097651 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 20:14:29.097665 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:14:29.097727 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 20:14:29.097752 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:14:29.097999 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0923 20:14:29.098107 (+ 108us) server_negotiation.cc:299] Negotiation successful
0923 20:14:29.098362 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":617,"thread_start_us":407,"threads_started":1}
I20260923 20:14:29.103324 1706 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 20:14:29.103516 23098 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2b165209a0ca415f928f259bdc8fb0d5: Bootstrap starting.
I20260923 20:14:29.106629 23098 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2b165209a0ca415f928f259bdc8fb0d5: Neither blocks nor log segments found. Creating new log.
I20260923 20:14:29.107707 23098 log.cc:824] T 00000000000000000000000000000000 P 2b165209a0ca415f928f259bdc8fb0d5: Log is configured to *not* fsync() on all Append() calls
I20260923 20:14:29.111454 23101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:14:29.108130 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.239:37899 (local address 127.0.0.1:37921)
0923 20:14:29.108409 (+ 279us) server_negotiation.cc:206] Beginning negotiation
0923 20:14:29.108419 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0923 20:14:29.109048 (+ 629us) server_negotiation.cc:407] Connection header received
0923 20:14:29.109258 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:14:29.109262 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:14:29.109311 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:14:29.109394 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:14:29.110478 (+ 1084us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:14:29.110484 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:14:29.110488 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 20:14:29.110506 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:14:29.110553 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 20:14:29.110584 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:14:29.110887 (+ 303us) server_negotiation.cc:1035] Waiting for connection context
0923 20:14:29.111042 (+ 155us) server_negotiation.cc:299] Negotiation successful
0923 20:14:29.111112 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260923 20:14:29.112226 23098 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2b165209a0ca415f928f259bdc8fb0d5: No bootstrap required, opened a new log
I20260923 20:14:29.121016 23098 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2b165209a0ca415f928f259bdc8fb0d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b165209a0ca415f928f259bdc8fb0d5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37921 } }
I20260923 20:14:29.121474 23098 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2b165209a0ca415f928f259bdc8fb0d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:14:29.121587 23098 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2b165209a0ca415f928f259bdc8fb0d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2b165209a0ca415f928f259bdc8fb0d5, State: Initialized, Role: FOLLOWER
I20260923 20:14:29.122417 23098 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2b165209a0ca415f928f259bdc8fb0d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b165209a0ca415f928f259bdc8fb0d5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37921 } }
I20260923 20:14:29.122702 23098 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2b165209a0ca415f928f259bdc8fb0d5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 20:14:29.122906 23098 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2b165209a0ca415f928f259bdc8fb0d5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 20:14:29.123103 23098 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2b165209a0ca415f928f259bdc8fb0d5 [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:14:29.125627 23098 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2b165209a0ca415f928f259bdc8fb0d5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b165209a0ca415f928f259bdc8fb0d5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37921 } }
I20260923 20:14:29.126302 23098 leader_election.cc:302] T 00000000000000000000000000000000 P 2b165209a0ca415f928f259bdc8fb0d5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2b165209a0ca415f928f259bdc8fb0d5; no voters:
I20260923 20:14:29.126950 23098 leader_election.cc:288] T 00000000000000000000000000000000 P 2b165209a0ca415f928f259bdc8fb0d5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 20:14:29.127101 23108 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2b165209a0ca415f928f259bdc8fb0d5 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 20:14:29.127831 23108 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2b165209a0ca415f928f259bdc8fb0d5 [term 1 LEADER]: Becoming Leader. State: Replica: 2b165209a0ca415f928f259bdc8fb0d5, State: Running, Role: LEADER
I20260923 20:14:29.128526 23108 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2b165209a0ca415f928f259bdc8fb0d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b165209a0ca415f928f259bdc8fb0d5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37921 } }
I20260923 20:14:29.129328 23098 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2b165209a0ca415f928f259bdc8fb0d5 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 20:14:29.131978 23109 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2b165209a0ca415f928f259bdc8fb0d5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2b165209a0ca415f928f259bdc8fb0d5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b165209a0ca415f928f259bdc8fb0d5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37921 } } }
I20260923 20:14:29.132521 23109 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2b165209a0ca415f928f259bdc8fb0d5 [sys.catalog]: This master's current role is: LEADER
I20260923 20:14:29.133172 23110 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2b165209a0ca415f928f259bdc8fb0d5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2b165209a0ca415f928f259bdc8fb0d5. Latest consensus state: current_term: 1 leader_uuid: "2b165209a0ca415f928f259bdc8fb0d5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b165209a0ca415f928f259bdc8fb0d5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37921 } } }
I20260923 20:14:29.133526 23110 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2b165209a0ca415f928f259bdc8fb0d5 [sys.catalog]: This master's current role is: LEADER
I20260923 20:14:29.135000 23117 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 20:14:29.140635 23117 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 20:14:29.149557 23117 catalog_manager.cc:1347] Generated new cluster ID: 5454609336d94d719e2834f4860756e1
I20260923 20:14:29.149699 23117 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 20:14:29.170063 23117 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 20:14:29.170624 23117 catalog_manager.cc:1504] Loading token signing keys...
I20260923 20:14:29.178992 23117 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2b165209a0ca415f928f259bdc8fb0d5: Generated new TSK 0
I20260923 20:14:29.180088 23117 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 20:14:29.186479 1706 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskqUuku6/build/asan/bin/kudu pid 23031
W20260923 20:14:29.187606 23093 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.239:59571
W20260923 20:14:29.188449 1706 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskqUuku6/build/asan/bin/kudu pid 23031: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260923 20:14:29.188598 1706 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqUuku6/build/asan/bin/kudu with pid 23031
2026-09-23T20:14:29Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1945 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790194121577027-1706-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 20:14:29 dist-test-slave-th4g krb5kdc[23129](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 20:14:29 dist-test-slave-th4g krb5kdc[23129](info): set up 2 sockets
Sep 23 20:14:29 dist-test-slave-th4g krb5kdc[23129](info): commencing operation
krb5kdc: starting...
W20260923 20:14:30.314509 1706 mini_kdc.cc:121] Time spent starting KDC: real 1.092s user 0.008s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790194121577027-1706-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790194121577027-1706-0/krb5kdc/test-user.keytab.
Sep 23 20:14:30 dist-test-slave-th4g krb5kdc[23129](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194470, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T20:14:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T20:14:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790194121577027-1706-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790194121577027-1706-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260923 20:14:30.606853 1706 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqUuku6/build/asan/bin/kudu
/tmp/dist-test-taskqUuku6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790194121577027-1706-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790194121577027-1706-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790194121577027-1706-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790194121577027-1706-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43693
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790194121577027-1706-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44993
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43693
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790194121577027-1706-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790194121577027-1706-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790194121577027-1706-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:14:30.983215 23142 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:14:30.983563 23142 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:14:30.983759 23142 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:14:30.991654 23142 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 20:14:30.991786 23142 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:14:30.991833 23142 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:14:30.991945 23142 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 20:14:30.992007 23142 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 20:14:31.002452 23142 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44993
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790194121577027-1706-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790194121577027-1706-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43693
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790194121577027-1706-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43693
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790194121577027-1706-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790194121577027-1706-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:14:31.003856 23142 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:14:31.005249 23142 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:14:31.019059 23148 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:14:31.019249 23147 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:14:31.019261 23150 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:14:31.019800 23142 server_base.cc:1034] running on GCE node
I20260923 20:14:31.020974 23142 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:14:31.022742 23142 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:14:31.024003 23142 hybrid_clock.cc:648] HybridClock initialized: now 1790194471023940 us; error 63 us; skew 500 ppm
I20260923 20:14:31.024448 23142 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:14:31 dist-test-slave-th4g krb5kdc[23129](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194471, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:14:31.034045 23142 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260923 20:14:31.036118 23142 webserver.cc:460] Webserver started at http://127.0.0.1:37197/ using document root <none> and password file <none>
I20260923 20:14:31.036900 23142 fs_manager.cc:359] Metadata directory not provided
I20260923 20:14:31.037053 23142 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:14:31.037384 23142 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:14:31.040679 23142 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790194121577027-1706-0/minicluster-data/master-0/data/instance:
uuid: "ee42b05a233c4e378fb626db4dbdea3b"
format_stamp: "Formatted at 2026-09-23 20:14:31 on dist-test-slave-th4g"
server_key: "c422b1381cc7ef2eb709514c274cd710"
server_key_iv: "b912d3fe44972796f58ab47e457ce83d"
server_key_version: "encryptionkey@0"
I20260923 20:14:31.041522 23142 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790194121577027-1706-0/minicluster-data/master-0/wal/instance:
uuid: "ee42b05a233c4e378fb626db4dbdea3b"
format_stamp: "Formatted at 2026-09-23 20:14:31 on dist-test-slave-th4g"
server_key: "c422b1381cc7ef2eb709514c274cd710"
server_key_iv: "b912d3fe44972796f58ab47e457ce83d"
server_key_version: "encryptionkey@0"
I20260923 20:14:31.047358 23142 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260923 20:14:31.051088 23157 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:14:31.053164 23142 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260923 20:14:31.053395 23142 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790194121577027-1706-0/minicluster-data/master-0/data,/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790194121577027-1706-0/minicluster-data/master-0/wal
uuid: "ee42b05a233c4e378fb626db4dbdea3b"
format_stamp: "Formatted at 2026-09-23 20:14:31 on dist-test-slave-th4g"
server_key: "c422b1381cc7ef2eb709514c274cd710"
server_key_iv: "b912d3fe44972796f58ab47e457ce83d"
server_key_version: "encryptionkey@0"
I20260923 20:14:31.053696 23142 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790194121577027-1706-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790194121577027-1706-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790194121577027-1706-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:14:31.096021 23142 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:14:31.102115 23142 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:14:31.102449 23142 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:14:31.126616 23142 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43693
I20260923 20:14:31.126645 23208 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43693 every 8 connection(s)
I20260923 20:14:31.128238 23142 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790194121577027-1706-0/minicluster-data/master-0/data/info.pb
I20260923 20:14:31.132556 23209 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:14:31.137387 1706 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqUuku6/build/asan/bin/kudu as pid 23142
I20260923 20:14:31.137614 1706 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790194121577027-1706-0/minicluster-data/master-0/wal/instance
I20260923 20:14:31.138525 1706 external_mini_cluster.cc:1428] Setting key ee089b1236edc5049d237b660d66fd3a
I20260923 20:14:31.145177 23209 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ee42b05a233c4e378fb626db4dbdea3b: Bootstrap starting.
I20260923 20:14:31.149578 23209 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ee42b05a233c4e378fb626db4dbdea3b: Neither blocks nor log segments found. Creating new log.
I20260923 20:14:31.151116 23209 log.cc:824] T 00000000000000000000000000000000 P ee42b05a233c4e378fb626db4dbdea3b: Log is configured to *not* fsync() on all Append() calls
I20260923 20:14:31.156128 23209 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ee42b05a233c4e378fb626db4dbdea3b: No bootstrap required, opened a new log
Sep 23 20:14:31 dist-test-slave-th4g krb5kdc[23129](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194470, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260923 20:14:31.167629 23209 raft_consensus.cc:348] T 00000000000000000000000000000000 P ee42b05a233c4e378fb626db4dbdea3b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee42b05a233c4e378fb626db4dbdea3b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43693 } }
I20260923 20:14:31.168056 23209 raft_consensus.cc:374] T 00000000000000000000000000000000 P ee42b05a233c4e378fb626db4dbdea3b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:14:31.168175 23209 raft_consensus.cc:729] T 00000000000000000000000000000000 P ee42b05a233c4e378fb626db4dbdea3b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee42b05a233c4e378fb626db4dbdea3b, State: Initialized, Role: FOLLOWER
I20260923 20:14:31.168956 23209 consensus_queue.cc:260] T 00000000000000000000000000000000 P ee42b05a233c4e378fb626db4dbdea3b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee42b05a233c4e378fb626db4dbdea3b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43693 } }
I20260923 20:14:31.169241 23209 raft_consensus.cc:388] T 00000000000000000000000000000000 P ee42b05a233c4e378fb626db4dbdea3b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 20:14:31.169386 23209 raft_consensus.cc:482] T 00000000000000000000000000000000 P ee42b05a233c4e378fb626db4dbdea3b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 20:14:31.169580 23209 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ee42b05a233c4e378fb626db4dbdea3b [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:14:31.170387 23212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:14:31.141155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.239:34269 (local address 127.0.0.1:43693)
0923 20:14:31.142462 (+ 1307us) server_negotiation.cc:206] Beginning negotiation
0923 20:14:31.142479 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0923 20:14:31.142541 (+ 62us) server_negotiation.cc:407] Connection header received
0923 20:14:31.144734 (+ 2193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:14:31.144772 (+ 38us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:14:31.145409 (+ 637us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:14:31.146011 (+ 602us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:14:31.148197 (+ 2186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:14:31.149679 (+ 1482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:14:31.151756 (+ 2077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:14:31.153658 (+ 1902us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:14:31.154008 (+ 350us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:14:31.159327 (+ 5319us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:14:31.159345 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:14:31.159371 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:14:31.159428 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:14:31.164183 (+ 4755us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:14:31.165315 (+ 1132us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:14:31.165325 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:14:31.165359 (+ 34us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:14:31.165643 (+ 284us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:14:31.166382 (+ 739us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:14:31.166388 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:14:31.166391 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:14:31.167195 (+ 804us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:14:31.167526 (+ 331us) server_negotiation.cc:1035] Waiting for connection context
0923 20:14:31.168454 (+ 928us) server_negotiation.cc:299] Negotiation successful
0923 20:14:31.168852 (+ 398us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":966,"thread_start_us":600,"threads_started":1}
I20260923 20:14:31.173062 23209 raft_consensus.cc:504] T 00000000000000000000000000000000 P ee42b05a233c4e378fb626db4dbdea3b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee42b05a233c4e378fb626db4dbdea3b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43693 } }
I20260923 20:14:31.173771 23209 leader_election.cc:302] T 00000000000000000000000000000000 P ee42b05a233c4e378fb626db4dbdea3b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ee42b05a233c4e378fb626db4dbdea3b; no voters:
I20260923 20:14:31.174496 23209 leader_election.cc:288] T 00000000000000000000000000000000 P ee42b05a233c4e378fb626db4dbdea3b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 20:14:31.174742 23214 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ee42b05a233c4e378fb626db4dbdea3b [term 1 FOLLOWER]: Leader election won for term 1
I20260923 20:14:31.177188 23214 raft_consensus.cc:686] T 00000000000000000000000000000000 P ee42b05a233c4e378fb626db4dbdea3b [term 1 LEADER]: Becoming Leader. State: Replica: ee42b05a233c4e378fb626db4dbdea3b, State: Running, Role: LEADER
I20260923 20:14:31.177531 1706 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 20:14:31.177918 23214 consensus_queue.cc:237] T 00000000000000000000000000000000 P ee42b05a233c4e378fb626db4dbdea3b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee42b05a233c4e378fb626db4dbdea3b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43693 } }
I20260923 20:14:31.178186 23209 sys_catalog.cc:564] T 00000000000000000000000000000000 P ee42b05a233c4e378fb626db4dbdea3b [sys.catalog]: configured and running, proceeding with master startup.
I20260923 20:14:31.181442 23215 sys_catalog.cc:455] T 00000000000000000000000000000000 P ee42b05a233c4e378fb626db4dbdea3b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ee42b05a233c4e378fb626db4dbdea3b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee42b05a233c4e378fb626db4dbdea3b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43693 } } }
I20260923 20:14:31.182061 23215 sys_catalog.cc:458] T 00000000000000000000000000000000 P ee42b05a233c4e378fb626db4dbdea3b [sys.catalog]: This master's current role is: LEADER
I20260923 20:14:31.186563 23216 sys_catalog.cc:455] T 00000000000000000000000000000000 P ee42b05a233c4e378fb626db4dbdea3b [sys.catalog]: SysCatalogTable state changed. Reason: New leader ee42b05a233c4e378fb626db4dbdea3b. Latest consensus state: current_term: 1 leader_uuid: "ee42b05a233c4e378fb626db4dbdea3b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee42b05a233c4e378fb626db4dbdea3b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43693 } } }
I20260923 20:14:31.187057 23216 sys_catalog.cc:458] T 00000000000000000000000000000000 P ee42b05a233c4e378fb626db4dbdea3b [sys.catalog]: This master's current role is: LEADER
I20260923 20:14:31.190312 23220 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 20:14:31.197543 23220 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 20:14:31.213914 23212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:14:31.191725 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.239:35511 (local address 127.0.0.1:43693)
0923 20:14:31.192191 (+ 466us) server_negotiation.cc:206] Beginning negotiation
0923 20:14:31.192198 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:14:31.192873 (+ 675us) server_negotiation.cc:407] Connection header received
0923 20:14:31.193181 (+ 308us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:14:31.193187 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:14:31.193268 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:14:31.193349 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:14:31.196283 (+ 2934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:14:31.197432 (+ 1149us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:14:31.199717 (+ 2285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:14:31.200412 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:14:31.200556 (+ 144us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:14:31.204730 (+ 4174us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:14:31.204738 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:14:31.204743 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:14:31.204804 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:14:31.209452 (+ 4648us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:14:31.210864 (+ 1412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:14:31.210870 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:14:31.210873 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:14:31.211040 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:14:31.211759 (+ 719us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:14:31.211764 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:14:31.211768 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:14:31.212234 (+ 466us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:14:31.212497 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0923 20:14:31.213258 (+ 761us) server_negotiation.cc:299] Negotiation successful
0923 20:14:31.213489 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":302}
I20260923 20:14:31.218245 23220 catalog_manager.cc:1347] Generated new cluster ID: 8e130c974d68423d8ad95f7d9f3c8c33
I20260923 20:14:31.218379 23220 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 20:14:31.254638 23220 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 20:14:31.256458 23220 catalog_manager.cc:1504] Loading token signing keys...
I20260923 20:14:31.281440 23220 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ee42b05a233c4e378fb626db4dbdea3b: Generated new TSK 0
I20260923 20:14:31.282680 23220 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 20:14:31.295210 1706 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskqUuku6/build/asan/bin/kudu pid 23142
I20260923 20:14:31.297451 23204 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:14:31.399570 1706 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqUuku6/build/asan/bin/kudu with pid 23142
W20260923 20:14:31.419932 23127 connection.cc:535] client connection to 127.0.0.1:43693 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:43693): Connection reset by peer (error 104)
2026-09-23T20:14:31Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (2221 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790194121577027-1706-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 20:14:31 dist-test-slave-th4g krb5kdc[23241](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 20:14:31 dist-test-slave-th4g krb5kdc[23241](info): set up 2 sockets
Sep 23 20:14:31 dist-test-slave-th4g krb5kdc[23241](info): commencing operation
krb5kdc: starting...
W20260923 20:14:32.550493 1706 mini_kdc.cc:121] Time spent starting KDC: real 1.106s user 0.007s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790194121577027-1706-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790194121577027-1706-0/krb5kdc/test-user.keytab.
Sep 23 20:14:32 dist-test-slave-th4g krb5kdc[23241](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194472, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T20:14:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T20:14:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790194121577027-1706-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790194121577027-1706-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260923 20:14:32.814431 1706 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqUuku6/build/asan/bin/kudu
/tmp/dist-test-taskqUuku6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790194121577027-1706-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790194121577027-1706-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790194121577027-1706-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790194121577027-1706-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45967
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790194121577027-1706-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40703
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45967
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790194121577027-1706-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790194121577027-1706-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790194121577027-1706-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:14:33.147804 23254 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:14:33.148093 23254 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:14:33.148257 23254 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:14:33.155580 23254 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 20:14:33.155706 23254 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:14:33.155794 23254 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:14:33.155906 23254 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 20:14:33.155969 23254 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 20:14:33.166627 23254 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40703
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790194121577027-1706-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790194121577027-1706-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45967
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790194121577027-1706-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45967
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790194121577027-1706-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790194121577027-1706-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:14:33.167963 23254 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:14:33.169273 23254 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:14:33.181599 23259 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:14:33.181644 23262 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:14:33.181675 23260 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:14:33.182722 23254 server_base.cc:1034] running on GCE node
I20260923 20:14:33.183550 23254 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:14:33.184991 23254 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:14:33.186221 23254 hybrid_clock.cc:648] HybridClock initialized: now 1790194473186180 us; error 67 us; skew 500 ppm
I20260923 20:14:33.186571 23254 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:14:33 dist-test-slave-th4g krb5kdc[23241](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194473, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:14:33.189963 23254 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260923 20:14:33.191478 23254 webserver.cc:460] Webserver started at http://127.0.0.1:37401/ using document root <none> and password file <none>
I20260923 20:14:33.192184 23254 fs_manager.cc:359] Metadata directory not provided
I20260923 20:14:33.192337 23254 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:14:33.192660 23254 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:14:33.195698 23254 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790194121577027-1706-0/minicluster-data/master-0/data/instance:
uuid: "ed28d7d260da4c58a9f01a56c34f76dc"
format_stamp: "Formatted at 2026-09-23 20:14:33 on dist-test-slave-th4g"
server_key: "55f655800019842dd733f456e1a8bbed"
server_key_iv: "48dce2dc0181cf981a9f99d48a3cfb16"
server_key_version: "encryptionkey@0"
I20260923 20:14:33.196480 23254 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790194121577027-1706-0/minicluster-data/master-0/wal/instance:
uuid: "ed28d7d260da4c58a9f01a56c34f76dc"
format_stamp: "Formatted at 2026-09-23 20:14:33 on dist-test-slave-th4g"
server_key: "55f655800019842dd733f456e1a8bbed"
server_key_iv: "48dce2dc0181cf981a9f99d48a3cfb16"
server_key_version: "encryptionkey@0"
I20260923 20:14:33.201733 23254 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260923 20:14:33.205221 23269 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:14:33.206617 23254 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 20:14:33.206808 23254 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790194121577027-1706-0/minicluster-data/master-0/data,/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790194121577027-1706-0/minicluster-data/master-0/wal
uuid: "ed28d7d260da4c58a9f01a56c34f76dc"
format_stamp: "Formatted at 2026-09-23 20:14:33 on dist-test-slave-th4g"
server_key: "55f655800019842dd733f456e1a8bbed"
server_key_iv: "48dce2dc0181cf981a9f99d48a3cfb16"
server_key_version: "encryptionkey@0"
I20260923 20:14:33.207007 23254 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790194121577027-1706-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790194121577027-1706-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790194121577027-1706-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:14:33.254500 23254 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:14:33.258810 23254 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:14:33.259099 23254 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:14:33.277984 23254 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45967
I20260923 20:14:33.277998 23320 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45967 every 8 connection(s)
I20260923 20:14:33.279323 23254 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790194121577027-1706-0/minicluster-data/master-0/data/info.pb
I20260923 20:14:33.279695 1706 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqUuku6/build/asan/bin/kudu as pid 23254
I20260923 20:14:33.279861 1706 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790194121577027-1706-0/minicluster-data/master-0/wal/instance
I20260923 20:14:33.280407 1706 external_mini_cluster.cc:1428] Setting key 7fdc7faa2a33ae07fd19de7ccb8291c7
I20260923 20:14:33.283694 23321 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:14:33.293035 23324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:14:33.282381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.239:54123 (local address 127.0.0.1:45967)
0923 20:14:33.283101 (+ 720us) server_negotiation.cc:206] Beginning negotiation
0923 20:14:33.283112 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0923 20:14:33.283151 (+ 39us) server_negotiation.cc:407] Connection header received
0923 20:14:33.284801 (+ 1650us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:14:33.284823 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:14:33.285213 (+ 390us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:14:33.285573 (+ 360us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:14:33.286133 (+ 560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:14:33.287438 (+ 1305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:14:33.288670 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:14:33.289331 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:14:33.289518 (+ 187us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:14:33.290777 (+ 1259us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:14:33.290785 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:14:33.290798 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 20:14:33.291339 (+ 541us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:14:33.291409 (+ 70us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 20:14:33.291437 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:14:33.291628 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0923 20:14:33.291933 (+ 305us) server_negotiation.cc:299] Negotiation successful
0923 20:14:33.292090 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":527,"thread_start_us":321,"threads_started":1}
I20260923 20:14:33.294520 23321 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ed28d7d260da4c58a9f01a56c34f76dc: Bootstrap starting.
I20260923 20:14:33.296880 1706 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 20:14:33.297736 23321 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ed28d7d260da4c58a9f01a56c34f76dc: Neither blocks nor log segments found. Creating new log.
I20260923 20:14:33.298740 23321 log.cc:824] T 00000000000000000000000000000000 P ed28d7d260da4c58a9f01a56c34f76dc: Log is configured to *not* fsync() on all Append() calls
I20260923 20:14:33.302083 23321 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ed28d7d260da4c58a9f01a56c34f76dc: No bootstrap required, opened a new log
I20260923 20:14:33.307436 23324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:14:33.301365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.239:43855 (local address 127.0.0.1:45967)
0923 20:14:33.301515 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0923 20:14:33.301520 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 20:14:33.302005 (+ 485us) server_negotiation.cc:407] Connection header received
0923 20:14:33.302224 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:14:33.302228 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:14:33.302293 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:14:33.302399 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:14:33.302888 (+ 489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:14:33.303653 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:14:33.304924 (+ 1271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:14:33.305391 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:14:33.305528 (+ 137us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:14:33.306707 (+ 1179us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:14:33.306712 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:14:33.306715 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 20:14:33.306735 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:14:33.306769 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 20:14:33.306788 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:14:33.306924 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0923 20:14:33.307117 (+ 193us) server_negotiation.cc:299] Negotiation successful
0923 20:14:33.307172 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260923 20:14:33.309952 23321 raft_consensus.cc:348] T 00000000000000000000000000000000 P ed28d7d260da4c58a9f01a56c34f76dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed28d7d260da4c58a9f01a56c34f76dc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45967 } }
I20260923 20:14:33.310408 23321 raft_consensus.cc:374] T 00000000000000000000000000000000 P ed28d7d260da4c58a9f01a56c34f76dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:14:33.310514 23321 raft_consensus.cc:729] T 00000000000000000000000000000000 P ed28d7d260da4c58a9f01a56c34f76dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed28d7d260da4c58a9f01a56c34f76dc, State: Initialized, Role: FOLLOWER
I20260923 20:14:33.311139 23321 consensus_queue.cc:260] T 00000000000000000000000000000000 P ed28d7d260da4c58a9f01a56c34f76dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed28d7d260da4c58a9f01a56c34f76dc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45967 } }
I20260923 20:14:33.311424 23321 raft_consensus.cc:388] T 00000000000000000000000000000000 P ed28d7d260da4c58a9f01a56c34f76dc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 20:14:33.311573 23321 raft_consensus.cc:482] T 00000000000000000000000000000000 P ed28d7d260da4c58a9f01a56c34f76dc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 20:14:33.311792 23321 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ed28d7d260da4c58a9f01a56c34f76dc [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:14:33.314533 23321 raft_consensus.cc:504] T 00000000000000000000000000000000 P ed28d7d260da4c58a9f01a56c34f76dc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed28d7d260da4c58a9f01a56c34f76dc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45967 } }
I20260923 20:14:33.315163 23321 leader_election.cc:302] T 00000000000000000000000000000000 P ed28d7d260da4c58a9f01a56c34f76dc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ed28d7d260da4c58a9f01a56c34f76dc; no voters:
I20260923 20:14:33.315820 23321 leader_election.cc:288] T 00000000000000000000000000000000 P ed28d7d260da4c58a9f01a56c34f76dc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 20:14:33.315992 23331 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ed28d7d260da4c58a9f01a56c34f76dc [term 1 FOLLOWER]: Leader election won for term 1
I20260923 20:14:33.316648 23331 raft_consensus.cc:686] T 00000000000000000000000000000000 P ed28d7d260da4c58a9f01a56c34f76dc [term 1 LEADER]: Becoming Leader. State: Replica: ed28d7d260da4c58a9f01a56c34f76dc, State: Running, Role: LEADER
I20260923 20:14:33.317131 23331 consensus_queue.cc:237] T 00000000000000000000000000000000 P ed28d7d260da4c58a9f01a56c34f76dc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed28d7d260da4c58a9f01a56c34f76dc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45967 } }
I20260923 20:14:33.317922 23321 sys_catalog.cc:564] T 00000000000000000000000000000000 P ed28d7d260da4c58a9f01a56c34f76dc [sys.catalog]: configured and running, proceeding with master startup.
I20260923 20:14:33.320082 23333 sys_catalog.cc:455] T 00000000000000000000000000000000 P ed28d7d260da4c58a9f01a56c34f76dc [sys.catalog]: SysCatalogTable state changed. Reason: New leader ed28d7d260da4c58a9f01a56c34f76dc. Latest consensus state: current_term: 1 leader_uuid: "ed28d7d260da4c58a9f01a56c34f76dc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed28d7d260da4c58a9f01a56c34f76dc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45967 } } }
I20260923 20:14:33.320379 23332 sys_catalog.cc:455] T 00000000000000000000000000000000 P ed28d7d260da4c58a9f01a56c34f76dc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ed28d7d260da4c58a9f01a56c34f76dc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed28d7d260da4c58a9f01a56c34f76dc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45967 } } }
I20260923 20:14:33.320590 23333 sys_catalog.cc:458] T 00000000000000000000000000000000 P ed28d7d260da4c58a9f01a56c34f76dc [sys.catalog]: This master's current role is: LEADER
I20260923 20:14:33.320664 23332 sys_catalog.cc:458] T 00000000000000000000000000000000 P ed28d7d260da4c58a9f01a56c34f76dc [sys.catalog]: This master's current role is: LEADER
I20260923 20:14:33.322741 23340 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 20:14:33.327782 23340 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 20:14:33.337248 23340 catalog_manager.cc:1347] Generated new cluster ID: 772b08e181d04a148f0250c401cb3578
I20260923 20:14:33.337424 23340 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 20:14:33.364437 23340 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 20:14:33.366505 23340 catalog_manager.cc:1504] Loading token signing keys...
I20260923 20:14:33.380730 23340 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ed28d7d260da4c58a9f01a56c34f76dc: Generated new TSK 0
I20260923 20:14:33.381769 23340 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 20:14:33.393950 1706 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskqUuku6/build/asan/bin/kudu pid 23254
W20260923 20:14:33.395107 23316 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.239:54123
W20260923 20:14:33.395844 1706 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskqUuku6/build/asan/bin/kudu pid 23254: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260923 20:14:33.396001 1706 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqUuku6/build/asan/bin/kudu with pid 23254
W20260923 20:14:33.413178 23239 connection.cc:535] client connection to 127.0.0.1:45967 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:45967): Connection reset by peer (error 104)
2026-09-23T20:14:33Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1987 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790194121577027-1706-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 20:14:33 dist-test-slave-th4g krb5kdc[23352](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 20:14:33 dist-test-slave-th4g krb5kdc[23352](info): set up 2 sockets
Sep 23 20:14:33 dist-test-slave-th4g krb5kdc[23352](info): commencing operation
krb5kdc: starting...
W20260923 20:14:34.518429 1706 mini_kdc.cc:121] Time spent starting KDC: real 1.086s user 0.005s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790194121577027-1706-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790194121577027-1706-0/krb5kdc/test-user.keytab.
Sep 23 20:14:34 dist-test-slave-th4g krb5kdc[23352](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194474, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T20:14:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T20:14:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790194121577027-1706-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790194121577027-1706-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260923 20:14:34.793804 1706 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqUuku6/build/asan/bin/kudu
/tmp/dist-test-taskqUuku6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790194121577027-1706-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790194121577027-1706-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790194121577027-1706-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790194121577027-1706-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41307
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790194121577027-1706-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37219
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41307
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790194121577027-1706-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790194121577027-1706-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790194121577027-1706-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:14:35.139743 23365 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:14:35.140036 23365 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:14:35.140218 23365 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:14:35.147642 23365 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 20:14:35.147768 23365 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:14:35.147890 23365 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:14:35.147979 23365 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 20:14:35.148063 23365 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 20:14:35.158205 23365 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37219
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790194121577027-1706-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790194121577027-1706-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41307
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790194121577027-1706-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41307
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790194121577027-1706-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790194121577027-1706-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:14:35.159571 23365 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:14:35.160878 23365 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:14:35.173381 23370 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:14:35.173476 23373 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:14:35.173476 23371 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:14:35.176126 23365 server_base.cc:1034] running on GCE node
I20260923 20:14:35.177155 23365 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:14:35.178834 23365 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:14:35.180080 23365 hybrid_clock.cc:648] HybridClock initialized: now 1790194475180013 us; error 54 us; skew 500 ppm
I20260923 20:14:35.180435 23365 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:14:35 dist-test-slave-th4g krb5kdc[23352](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194475, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:14:35.183871 23365 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260923 20:14:35.185461 23365 webserver.cc:460] Webserver started at http://127.0.0.1:44147/ using document root <none> and password file <none>
I20260923 20:14:35.186245 23365 fs_manager.cc:359] Metadata directory not provided
I20260923 20:14:35.186352 23365 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:14:35.186599 23365 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:14:35.189587 23365 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790194121577027-1706-0/minicluster-data/master-0/data/instance:
uuid: "8e8965f9b6454a55a2d2f58fecf5ae37"
format_stamp: "Formatted at 2026-09-23 20:14:35 on dist-test-slave-th4g"
server_key: "65b84445f7482a02254c09c9bb6d589e"
server_key_iv: "2aa090b5705126aeb455512a32304844"
server_key_version: "encryptionkey@0"
I20260923 20:14:35.190436 23365 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790194121577027-1706-0/minicluster-data/master-0/wal/instance:
uuid: "8e8965f9b6454a55a2d2f58fecf5ae37"
format_stamp: "Formatted at 2026-09-23 20:14:35 on dist-test-slave-th4g"
server_key: "65b84445f7482a02254c09c9bb6d589e"
server_key_iv: "2aa090b5705126aeb455512a32304844"
server_key_version: "encryptionkey@0"
I20260923 20:14:35.196009 23365 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260923 20:14:35.199645 23380 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:14:35.201242 23365 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260923 20:14:35.201450 23365 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790194121577027-1706-0/minicluster-data/master-0/data,/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790194121577027-1706-0/minicluster-data/master-0/wal
uuid: "8e8965f9b6454a55a2d2f58fecf5ae37"
format_stamp: "Formatted at 2026-09-23 20:14:35 on dist-test-slave-th4g"
server_key: "65b84445f7482a02254c09c9bb6d589e"
server_key_iv: "2aa090b5705126aeb455512a32304844"
server_key_version: "encryptionkey@0"
I20260923 20:14:35.201663 23365 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790194121577027-1706-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790194121577027-1706-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790194121577027-1706-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:14:35.207738 23365 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:14:35.212347 23365 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:14:35.212685 23365 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:14:35.232018 23365 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41307
I20260923 20:14:35.232100 23431 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41307 every 8 connection(s)
I20260923 20:14:35.233455 23365 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790194121577027-1706-0/minicluster-data/master-0/data/info.pb
I20260923 20:14:35.237517 23432 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:14:35.241248 1706 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqUuku6/build/asan/bin/kudu as pid 23365
I20260923 20:14:35.241442 1706 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790194121577027-1706-0/minicluster-data/master-0/wal/instance
I20260923 20:14:35.241932 1706 external_mini_cluster.cc:1428] Setting key 4f926e6fdd6200280f6623e3914772b4
I20260923 20:14:35.250080 23435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:14:35.243891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.239:55225 (local address 127.0.0.1:41307)
0923 20:14:35.244691 (+ 800us) server_negotiation.cc:206] Beginning negotiation
0923 20:14:35.244702 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0923 20:14:35.244744 (+ 42us) server_negotiation.cc:407] Connection header received
0923 20:14:35.246468 (+ 1724us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:14:35.246491 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:14:35.246871 (+ 380us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:14:35.247230 (+ 359us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:14:35.248390 (+ 1160us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:14:35.248396 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:14:35.248409 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 20:14:35.248423 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:14:35.248484 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 20:14:35.248510 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:14:35.248682 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0923 20:14:35.248792 (+ 110us) server_negotiation.cc:299] Negotiation successful
0923 20:14:35.249067 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":587,"thread_start_us":372,"threads_started":1}
I20260923 20:14:35.250922 23432 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8e8965f9b6454a55a2d2f58fecf5ae37: Bootstrap starting.
I20260923 20:14:35.254752 23432 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8e8965f9b6454a55a2d2f58fecf5ae37: Neither blocks nor log segments found. Creating new log.
I20260923 20:14:35.255234 1706 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 20:14:35.256173 23432 log.cc:824] T 00000000000000000000000000000000 P 8e8965f9b6454a55a2d2f58fecf5ae37: Log is configured to *not* fsync() on all Append() calls
I20260923 20:14:35.259922 23432 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8e8965f9b6454a55a2d2f58fecf5ae37: No bootstrap required, opened a new log
I20260923 20:14:35.263221 23435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:14:35.260507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.239:41117 (local address 127.0.0.1:41307)
0923 20:14:35.260731 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0923 20:14:35.260739 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 20:14:35.261036 (+ 297us) server_negotiation.cc:407] Connection header received
0923 20:14:35.261257 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:14:35.261261 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:14:35.261311 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:14:35.261417 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:14:35.262462 (+ 1045us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:14:35.262468 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:14:35.262473 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 20:14:35.262496 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:14:35.262548 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 20:14:35.262563 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:14:35.262768 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0923 20:14:35.262862 (+ 94us) server_negotiation.cc:299] Negotiation successful
0923 20:14:35.262933 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260923 20:14:35.268332 23432 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8e8965f9b6454a55a2d2f58fecf5ae37 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e8965f9b6454a55a2d2f58fecf5ae37" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41307 } }
I20260923 20:14:35.268718 23432 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8e8965f9b6454a55a2d2f58fecf5ae37 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:14:35.268847 23432 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8e8965f9b6454a55a2d2f58fecf5ae37 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8e8965f9b6454a55a2d2f58fecf5ae37, State: Initialized, Role: FOLLOWER
I20260923 20:14:35.269462 23432 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8e8965f9b6454a55a2d2f58fecf5ae37 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e8965f9b6454a55a2d2f58fecf5ae37" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41307 } }
I20260923 20:14:35.269667 23432 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8e8965f9b6454a55a2d2f58fecf5ae37 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 20:14:35.269824 23432 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8e8965f9b6454a55a2d2f58fecf5ae37 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 20:14:35.270066 23432 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8e8965f9b6454a55a2d2f58fecf5ae37 [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:14:35.272452 23432 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8e8965f9b6454a55a2d2f58fecf5ae37 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e8965f9b6454a55a2d2f58fecf5ae37" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41307 } }
I20260923 20:14:35.273074 23432 leader_election.cc:302] T 00000000000000000000000000000000 P 8e8965f9b6454a55a2d2f58fecf5ae37 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8e8965f9b6454a55a2d2f58fecf5ae37; no voters:
I20260923 20:14:35.273736 23432 leader_election.cc:288] T 00000000000000000000000000000000 P 8e8965f9b6454a55a2d2f58fecf5ae37 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 20:14:35.273934 23442 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8e8965f9b6454a55a2d2f58fecf5ae37 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 20:14:35.274668 23442 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8e8965f9b6454a55a2d2f58fecf5ae37 [term 1 LEADER]: Becoming Leader. State: Replica: 8e8965f9b6454a55a2d2f58fecf5ae37, State: Running, Role: LEADER
I20260923 20:14:35.275264 23442 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8e8965f9b6454a55a2d2f58fecf5ae37 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e8965f9b6454a55a2d2f58fecf5ae37" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41307 } }
I20260923 20:14:35.276022 23432 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8e8965f9b6454a55a2d2f58fecf5ae37 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 20:14:35.277933 23443 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8e8965f9b6454a55a2d2f58fecf5ae37 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8e8965f9b6454a55a2d2f58fecf5ae37" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e8965f9b6454a55a2d2f58fecf5ae37" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41307 } } }
I20260923 20:14:35.278366 23443 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8e8965f9b6454a55a2d2f58fecf5ae37 [sys.catalog]: This master's current role is: LEADER
I20260923 20:14:35.279711 23444 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8e8965f9b6454a55a2d2f58fecf5ae37 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8e8965f9b6454a55a2d2f58fecf5ae37. Latest consensus state: current_term: 1 leader_uuid: "8e8965f9b6454a55a2d2f58fecf5ae37" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e8965f9b6454a55a2d2f58fecf5ae37" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41307 } } }
I20260923 20:14:35.280078 23444 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8e8965f9b6454a55a2d2f58fecf5ae37 [sys.catalog]: This master's current role is: LEADER
I20260923 20:14:35.283674 23448 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 20:14:35.289346 23448 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 20:14:35.298709 23448 catalog_manager.cc:1347] Generated new cluster ID: 4588b52d49ba4bb5950d2449c5f5ab1d
I20260923 20:14:35.298859 23448 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 20:14:35.342526 23448 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 20:14:35.343070 23448 catalog_manager.cc:1504] Loading token signing keys...
I20260923 20:14:35.354003 23448 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8e8965f9b6454a55a2d2f58fecf5ae37: Generated new TSK 0
I20260923 20:14:35.354984 23448 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 20:14:35.361428 1706 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskqUuku6/build/asan/bin/kudu pid 23365
W20260923 20:14:35.362619 23427 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.239:55225
W20260923 20:14:35.363629 1706 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskqUuku6/build/asan/bin/kudu pid 23365: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260923 20:14:35.363854 1706 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqUuku6/build/asan/bin/kudu with pid 23365
2026-09-23T20:14:35Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1969 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790194121577027-1706-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 20:14:35 dist-test-slave-th4g krb5kdc[23463](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 20:14:35 dist-test-slave-th4g krb5kdc[23463](info): set up 2 sockets
Sep 23 20:14:35 dist-test-slave-th4g krb5kdc[23463](info): commencing operation
krb5kdc: starting...
W20260923 20:14:36.498747 1706 mini_kdc.cc:121] Time spent starting KDC: real 1.097s user 0.000s sys 0.031s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790194121577027-1706-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790194121577027-1706-0/krb5kdc/test-user.keytab.
Sep 23 20:14:36 dist-test-slave-th4g krb5kdc[23463](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194476, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T20:14:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T20:14:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.170.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.170.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.170.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790194121577027-1706-0/krb5kdc/kudu_127.1.170.190.keytab.
Entry for principal kudu/127.1.170.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790194121577027-1706-0/krb5kdc/kudu_127.1.170.190.keytab.
I20260923 20:14:36.760900 1706 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqUuku6/build/asan/bin/kudu
/tmp/dist-test-taskqUuku6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790194121577027-1706-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790194121577027-1706-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790194121577027-1706-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790194121577027-1706-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.170.190:41633
--webserver_interface=127.1.170.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790194121577027-1706-0/krb5kdc/kudu_127.1.170.190.keytab
--principal=kudu/127.1.170.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.170.148:36757
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.170.190:41633
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790194121577027-1706-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790194121577027-1706-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790194121577027-1706-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:14:37.102658 23476 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:14:37.103142 23476 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:14:37.103353 23476 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:14:37.114424 23476 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 20:14:37.114562 23476 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:14:37.114652 23476 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:14:37.114760 23476 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 20:14:37.114826 23476 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 20:14:37.125463 23476 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.170.148:36757
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790194121577027-1706-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790194121577027-1706-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.170.190:41633
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790194121577027-1706-0/krb5kdc/kudu_127.1.170.190.keytab
--principal=kudu/127.1.170.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.170.190:41633
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790194121577027-1706-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.170.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790194121577027-1706-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:14:37.126818 23476 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:14:37.128078 23476 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:14:37.140445 23481 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:14:37.140595 23484 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:14:37.140431 23482 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:14:37.141983 23476 server_base.cc:1034] running on GCE node
I20260923 20:14:37.142805 23476 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:14:37.144176 23476 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:14:37.145390 23476 hybrid_clock.cc:648] HybridClock initialized: now 1790194477145345 us; error 41 us; skew 500 ppm
I20260923 20:14:37.145747 23476 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:14:37 dist-test-slave-th4g krb5kdc[23463](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194477, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.170.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:14:37.149004 23476 init.cc:377] Logged in from keytab as kudu/127.1.170.190@KRBTEST.COM (short username kudu)
I20260923 20:14:37.150645 23476 webserver.cc:460] Webserver started at http://127.1.170.190:33675/ using document root <none> and password file <none>
I20260923 20:14:37.151329 23476 fs_manager.cc:359] Metadata directory not provided
I20260923 20:14:37.151458 23476 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:14:37.151791 23476 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:14:37.155090 23476 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790194121577027-1706-0/minicluster-data/master-0/data/instance:
uuid: "9203c3eb5bef4cb3b767a9b5149ad89a"
format_stamp: "Formatted at 2026-09-23 20:14:37 on dist-test-slave-th4g"
server_key: "affd542a4a56f1302d5b53a190175937"
server_key_iv: "fa427b2f5b4f4d4241435336edc61c1b"
server_key_version: "encryptionkey@0"
I20260923 20:14:37.155897 23476 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790194121577027-1706-0/minicluster-data/master-0/wal/instance:
uuid: "9203c3eb5bef4cb3b767a9b5149ad89a"
format_stamp: "Formatted at 2026-09-23 20:14:37 on dist-test-slave-th4g"
server_key: "affd542a4a56f1302d5b53a190175937"
server_key_iv: "fa427b2f5b4f4d4241435336edc61c1b"
server_key_version: "encryptionkey@0"
I20260923 20:14:37.161180 23476 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.003s
I20260923 20:14:37.164791 23491 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:14:37.166425 23476 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 20:14:37.166613 23476 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790194121577027-1706-0/minicluster-data/master-0/data,/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790194121577027-1706-0/minicluster-data/master-0/wal
uuid: "9203c3eb5bef4cb3b767a9b5149ad89a"
format_stamp: "Formatted at 2026-09-23 20:14:37 on dist-test-slave-th4g"
server_key: "affd542a4a56f1302d5b53a190175937"
server_key_iv: "fa427b2f5b4f4d4241435336edc61c1b"
server_key_version: "encryptionkey@0"
I20260923 20:14:37.166822 23476 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790194121577027-1706-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790194121577027-1706-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790194121577027-1706-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:14:37.187108 23476 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:14:37.191252 23476 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:14:37.191537 23476 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:14:37.211036 23476 rpc_server.cc:307] RPC server started. Bound to: 127.1.170.190:41633
I20260923 20:14:37.211058 23542 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.170.190:41633 every 8 connection(s)
I20260923 20:14:37.212518 23476 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790194121577027-1706-0/minicluster-data/master-0/data/info.pb
I20260923 20:14:37.216327 23543 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:14:37.216423 1706 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqUuku6/build/asan/bin/kudu as pid 23476
I20260923 20:14:37.216645 1706 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790194121577027-1706-0/minicluster-data/master-0/wal/instance
I20260923 20:14:37.217119 1706 external_mini_cluster.cc:1428] Setting key 85d77e00607cdb1a0771798bba3d731d
I20260923 20:14:37.231350 23543 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9203c3eb5bef4cb3b767a9b5149ad89a: Bootstrap starting.
Sep 23 20:14:37 dist-test-slave-th4g krb5kdc[23463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194476, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.170.190@KRBTEST.COM
I20260923 20:14:37.235239 23543 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9203c3eb5bef4cb3b767a9b5149ad89a: Neither blocks nor log segments found. Creating new log.
I20260923 20:14:37.236411 23543 log.cc:824] T 00000000000000000000000000000000 P 9203c3eb5bef4cb3b767a9b5149ad89a: Log is configured to *not* fsync() on all Append() calls
I20260923 20:14:37.239739 23543 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9203c3eb5bef4cb3b767a9b5149ad89a: No bootstrap required, opened a new log
I20260923 20:14:37.240751 23546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:14:37.219033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43289 (local address 127.1.170.190:41633)
0923 20:14:37.219932 (+ 899us) server_negotiation.cc:206] Beginning negotiation
0923 20:14:37.219947 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0923 20:14:37.220001 (+ 54us) server_negotiation.cc:407] Connection header received
0923 20:14:37.221847 (+ 1846us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:14:37.221907 (+ 60us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:14:37.222471 (+ 564us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:14:37.222942 (+ 471us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:14:37.224194 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:14:37.226181 (+ 1987us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:14:37.227471 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:14:37.228565 (+ 1094us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:14:37.228750 (+ 185us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:14:37.233307 (+ 4557us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:14:37.233315 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:14:37.233329 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:14:37.233364 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:14:37.236541 (+ 3177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:14:37.237460 (+ 919us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:14:37.237470 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:14:37.237482 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:14:37.237708 (+ 226us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:14:37.238255 (+ 547us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:14:37.238258 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:14:37.238260 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:14:37.238699 (+ 439us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:14:37.238928 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0923 20:14:37.239453 (+ 525us) server_negotiation.cc:299] Negotiation successful
0923 20:14:37.239748 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":655,"thread_start_us":402,"threads_started":1}
I20260923 20:14:37.244594 1706 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 20:14:37.249460 23543 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9203c3eb5bef4cb3b767a9b5149ad89a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9203c3eb5bef4cb3b767a9b5149ad89a" member_type: VOTER last_known_addr { host: "127.1.170.190" port: 41633 } }
I20260923 20:14:37.249977 23543 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9203c3eb5bef4cb3b767a9b5149ad89a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:14:37.250101 23543 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9203c3eb5bef4cb3b767a9b5149ad89a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9203c3eb5bef4cb3b767a9b5149ad89a, State: Initialized, Role: FOLLOWER
I20260923 20:14:37.250833 23543 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9203c3eb5bef4cb3b767a9b5149ad89a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9203c3eb5bef4cb3b767a9b5149ad89a" member_type: VOTER last_known_addr { host: "127.1.170.190" port: 41633 } }
I20260923 20:14:37.251166 23543 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9203c3eb5bef4cb3b767a9b5149ad89a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 20:14:37.251328 23543 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9203c3eb5bef4cb3b767a9b5149ad89a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 20:14:37.251515 23543 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9203c3eb5bef4cb3b767a9b5149ad89a [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:14:37.254739 23543 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9203c3eb5bef4cb3b767a9b5149ad89a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9203c3eb5bef4cb3b767a9b5149ad89a" member_type: VOTER last_known_addr { host: "127.1.170.190" port: 41633 } }
I20260923 20:14:37.255358 23543 leader_election.cc:302] T 00000000000000000000000000000000 P 9203c3eb5bef4cb3b767a9b5149ad89a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9203c3eb5bef4cb3b767a9b5149ad89a; no voters:
I20260923 20:14:37.256127 23543 leader_election.cc:288] T 00000000000000000000000000000000 P 9203c3eb5bef4cb3b767a9b5149ad89a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 20:14:37.256269 23553 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9203c3eb5bef4cb3b767a9b5149ad89a [term 1 FOLLOWER]: Leader election won for term 1
I20260923 20:14:37.256949 23553 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9203c3eb5bef4cb3b767a9b5149ad89a [term 1 LEADER]: Becoming Leader. State: Replica: 9203c3eb5bef4cb3b767a9b5149ad89a, State: Running, Role: LEADER
I20260923 20:14:37.258158 23553 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9203c3eb5bef4cb3b767a9b5149ad89a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9203c3eb5bef4cb3b767a9b5149ad89a" member_type: VOTER last_known_addr { host: "127.1.170.190" port: 41633 } }
I20260923 20:14:37.261422 23543 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9203c3eb5bef4cb3b767a9b5149ad89a [sys.catalog]: configured and running, proceeding with master startup.
I20260923 20:14:37.261861 23554 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9203c3eb5bef4cb3b767a9b5149ad89a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9203c3eb5bef4cb3b767a9b5149ad89a. Latest consensus state: current_term: 1 leader_uuid: "9203c3eb5bef4cb3b767a9b5149ad89a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9203c3eb5bef4cb3b767a9b5149ad89a" member_type: VOTER last_known_addr { host: "127.1.170.190" port: 41633 } } }
I20260923 20:14:37.262295 23554 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9203c3eb5bef4cb3b767a9b5149ad89a [sys.catalog]: This master's current role is: LEADER
I20260923 20:14:37.268945 23546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:14:37.250000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57751 (local address 127.1.170.190:41633)
0923 20:14:37.250239 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0923 20:14:37.250249 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0923 20:14:37.250734 (+ 485us) server_negotiation.cc:407] Connection header received
0923 20:14:37.250901 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:14:37.250905 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:14:37.250968 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:14:37.251058 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:14:37.253160 (+ 2102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:14:37.254047 (+ 887us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:14:37.255772 (+ 1725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:14:37.256322 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:14:37.256478 (+ 156us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:14:37.259396 (+ 2918us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:14:37.259405 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:14:37.259412 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:14:37.259461 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:14:37.263740 (+ 4279us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:14:37.265104 (+ 1364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:14:37.265112 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:14:37.265118 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:14:37.265337 (+ 219us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:14:37.266088 (+ 751us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:14:37.266095 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:14:37.266101 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:14:37.266558 (+ 457us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:14:37.268042 (+ 1484us) server_negotiation.cc:1035] Waiting for connection context
0923 20:14:37.268277 (+ 235us) server_negotiation.cc:299] Negotiation successful
0923 20:14:37.268503 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260923 20:14:37.269946 23555 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9203c3eb5bef4cb3b767a9b5149ad89a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9203c3eb5bef4cb3b767a9b5149ad89a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9203c3eb5bef4cb3b767a9b5149ad89a" member_type: VOTER last_known_addr { host: "127.1.170.190" port: 41633 } } }
I20260923 20:14:37.270381 23555 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9203c3eb5bef4cb3b767a9b5149ad89a [sys.catalog]: This master's current role is: LEADER
I20260923 20:14:37.276360 23557 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 20:14:37.280469 23557 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 20:14:37.288993 23557 catalog_manager.cc:1347] Generated new cluster ID: e858853c352e4b0fb2f30e2bf41f4fce
I20260923 20:14:37.289119 23557 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 20:14:37.318396 23557 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 20:14:37.320271 23557 catalog_manager.cc:1504] Loading token signing keys...
I20260923 20:14:37.327736 23557 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9203c3eb5bef4cb3b767a9b5149ad89a: Generated new TSK 0
I20260923 20:14:37.328812 23557 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 20:14:37.335563 1706 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskqUuku6/build/asan/bin/kudu pid 23476
I20260923 20:14:37.336764 23538 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:14:37.430521 1706 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqUuku6/build/asan/bin/kudu with pid 23476
W20260923 20:14:37.448153 23461 connection.cc:535] client connection to 127.1.170.190:41633 recv error: Network error: failed to read from TLS socket (remote: 127.1.170.190:41633): Connection reset by peer (error 104)
2026-09-23T20:14:37Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (2065 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790194121577027-1706-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 20:14:37 dist-test-slave-th4g krb5kdc[23575](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 20:14:37 dist-test-slave-th4g krb5kdc[23575](info): set up 2 sockets
Sep 23 20:14:37 dist-test-slave-th4g krb5kdc[23575](info): commencing operation
krb5kdc: starting...
W20260923 20:14:38.542747 1706 mini_kdc.cc:121] Time spent starting KDC: real 1.075s user 0.001s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790194121577027-1706-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790194121577027-1706-0/krb5kdc/test-user.keytab.
Sep 23 20:14:38 dist-test-slave-th4g krb5kdc[23575](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194478, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T20:14:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T20:14:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.170.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.170.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.170.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790194121577027-1706-0/krb5kdc/kudu_127.1.170.190.keytab.
Entry for principal kudu/127.1.170.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790194121577027-1706-0/krb5kdc/kudu_127.1.170.190.keytab.
I20260923 20:14:38.830675 1706 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqUuku6/build/asan/bin/kudu
/tmp/dist-test-taskqUuku6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790194121577027-1706-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790194121577027-1706-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790194121577027-1706-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790194121577027-1706-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.170.190:38203
--webserver_interface=127.1.170.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790194121577027-1706-0/krb5kdc/kudu_127.1.170.190.keytab
--principal=kudu/127.1.170.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.170.148:45341
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.170.190:38203
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790194121577027-1706-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790194121577027-1706-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790194121577027-1706-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:14:39.178256 23588 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:14:39.178553 23588 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:14:39.178654 23588 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:14:39.186877 23588 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 20:14:39.187013 23588 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:14:39.187063 23588 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:14:39.187125 23588 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 20:14:39.187171 23588 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 20:14:39.198076 23588 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.170.148:45341
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790194121577027-1706-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790194121577027-1706-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.170.190:38203
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790194121577027-1706-0/krb5kdc/kudu_127.1.170.190.keytab
--principal=kudu/127.1.170.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.170.190:38203
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790194121577027-1706-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.170.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790194121577027-1706-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:14:39.199499 23588 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:14:39.200748 23588 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:14:39.213171 23593 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:14:39.213239 23594 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:14:39.213588 23596 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:14:39.213960 23588 server_base.cc:1034] running on GCE node
I20260923 20:14:39.214741 23588 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:14:39.216162 23588 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:14:39.217383 23588 hybrid_clock.cc:648] HybridClock initialized: now 1790194479217341 us; error 33 us; skew 500 ppm
I20260923 20:14:39.217718 23588 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:14:39 dist-test-slave-th4g krb5kdc[23575](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194479, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.170.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:14:39.220999 23588 init.cc:377] Logged in from keytab as kudu/127.1.170.190@KRBTEST.COM (short username kudu)
I20260923 20:14:39.222618 23588 webserver.cc:460] Webserver started at http://127.1.170.190:34785/ using document root <none> and password file <none>
I20260923 20:14:39.223306 23588 fs_manager.cc:359] Metadata directory not provided
I20260923 20:14:39.223438 23588 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:14:39.223747 23588 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:14:39.227172 23588 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790194121577027-1706-0/minicluster-data/master-0/data/instance:
uuid: "49ab34c3f207402cbd179b60f0497365"
format_stamp: "Formatted at 2026-09-23 20:14:39 on dist-test-slave-th4g"
server_key: "2f6fcb846df4215ad78b40e416b9a74d"
server_key_iv: "03e588849ecbba9fdcd36b569a37917e"
server_key_version: "encryptionkey@0"
I20260923 20:14:39.227986 23588 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790194121577027-1706-0/minicluster-data/master-0/wal/instance:
uuid: "49ab34c3f207402cbd179b60f0497365"
format_stamp: "Formatted at 2026-09-23 20:14:39 on dist-test-slave-th4g"
server_key: "2f6fcb846df4215ad78b40e416b9a74d"
server_key_iv: "03e588849ecbba9fdcd36b569a37917e"
server_key_version: "encryptionkey@0"
I20260923 20:14:39.234210 23588 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.004s
I20260923 20:14:39.239295 23603 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:14:39.240983 23588 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.001s
I20260923 20:14:39.241221 23588 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790194121577027-1706-0/minicluster-data/master-0/data,/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790194121577027-1706-0/minicluster-data/master-0/wal
uuid: "49ab34c3f207402cbd179b60f0497365"
format_stamp: "Formatted at 2026-09-23 20:14:39 on dist-test-slave-th4g"
server_key: "2f6fcb846df4215ad78b40e416b9a74d"
server_key_iv: "03e588849ecbba9fdcd36b569a37917e"
server_key_version: "encryptionkey@0"
I20260923 20:14:39.241468 23588 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790194121577027-1706-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790194121577027-1706-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790194121577027-1706-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:14:39.264793 23588 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:14:39.270176 23588 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:14:39.270505 23588 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:14:39.292734 23588 rpc_server.cc:307] RPC server started. Bound to: 127.1.170.190:38203
I20260923 20:14:39.292862 23654 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.170.190:38203 every 8 connection(s)
I20260923 20:14:39.294435 23588 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790194121577027-1706-0/minicluster-data/master-0/data/info.pb
I20260923 20:14:39.298430 23655 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:14:39.298485 1706 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqUuku6/build/asan/bin/kudu as pid 23588
I20260923 20:14:39.298681 1706 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790194121577027-1706-0/minicluster-data/master-0/wal/instance
I20260923 20:14:39.299124 1706 external_mini_cluster.cc:1428] Setting key 0545e1ae47de0b70fda16ace3c938d67
I20260923 20:14:39.312448 23655 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 49ab34c3f207402cbd179b60f0497365: Bootstrap starting.
I20260923 20:14:39.312987 23658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:14:39.300907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41309 (local address 127.1.170.190:38203)
0923 20:14:39.301792 (+ 885us) server_negotiation.cc:206] Beginning negotiation
0923 20:14:39.301803 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0923 20:14:39.301843 (+ 40us) server_negotiation.cc:407] Connection header received
0923 20:14:39.303393 (+ 1550us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:14:39.303415 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:14:39.303828 (+ 413us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:14:39.304152 (+ 324us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:14:39.304763 (+ 611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:14:39.306307 (+ 1544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:14:39.308229 (+ 1922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:14:39.308948 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:14:39.309097 (+ 149us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:14:39.310503 (+ 1406us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:14:39.310511 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:14:39.310526 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 20:14:39.311147 (+ 621us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:14:39.311233 (+ 86us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 20:14:39.311258 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:14:39.311384 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0923 20:14:39.311750 (+ 366us) server_negotiation.cc:299] Negotiation successful
0923 20:14:39.311946 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":662,"thread_start_us":428,"threads_started":1}
I20260923 20:14:39.316104 23655 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 49ab34c3f207402cbd179b60f0497365: Neither blocks nor log segments found. Creating new log.
I20260923 20:14:39.317111 1706 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 20:14:39.317302 23655 log.cc:824] T 00000000000000000000000000000000 P 49ab34c3f207402cbd179b60f0497365: Log is configured to *not* fsync() on all Append() calls
I20260923 20:14:39.320003 23655 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 49ab34c3f207402cbd179b60f0497365: No bootstrap required, opened a new log
I20260923 20:14:39.328939 23658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:14:39.321587 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60581 (local address 127.1.170.190:38203)
0923 20:14:39.321790 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0923 20:14:39.321796 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:14:39.322327 (+ 531us) server_negotiation.cc:407] Connection header received
0923 20:14:39.322490 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:14:39.322496 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:14:39.322576 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:14:39.322711 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:14:39.323252 (+ 541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:14:39.324446 (+ 1194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:14:39.325789 (+ 1343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:14:39.326567 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:14:39.326738 (+ 171us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:14:39.328070 (+ 1332us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:14:39.328079 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:14:39.328084 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 20:14:39.328115 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:14:39.328163 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 20:14:39.328185 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:14:39.328302 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0923 20:14:39.328515 (+ 213us) server_negotiation.cc:299] Negotiation successful
0923 20:14:39.328571 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260923 20:14:39.329986 23655 raft_consensus.cc:348] T 00000000000000000000000000000000 P 49ab34c3f207402cbd179b60f0497365 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49ab34c3f207402cbd179b60f0497365" member_type: VOTER last_known_addr { host: "127.1.170.190" port: 38203 } }
I20260923 20:14:39.330405 23655 raft_consensus.cc:374] T 00000000000000000000000000000000 P 49ab34c3f207402cbd179b60f0497365 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:14:39.330546 23655 raft_consensus.cc:729] T 00000000000000000000000000000000 P 49ab34c3f207402cbd179b60f0497365 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49ab34c3f207402cbd179b60f0497365, State: Initialized, Role: FOLLOWER
I20260923 20:14:39.331159 23655 consensus_queue.cc:260] T 00000000000000000000000000000000 P 49ab34c3f207402cbd179b60f0497365 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49ab34c3f207402cbd179b60f0497365" member_type: VOTER last_known_addr { host: "127.1.170.190" port: 38203 } }
I20260923 20:14:39.331373 23655 raft_consensus.cc:388] T 00000000000000000000000000000000 P 49ab34c3f207402cbd179b60f0497365 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 20:14:39.331498 23655 raft_consensus.cc:482] T 00000000000000000000000000000000 P 49ab34c3f207402cbd179b60f0497365 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 20:14:39.331687 23655 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 49ab34c3f207402cbd179b60f0497365 [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:14:39.334020 23655 raft_consensus.cc:504] T 00000000000000000000000000000000 P 49ab34c3f207402cbd179b60f0497365 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49ab34c3f207402cbd179b60f0497365" member_type: VOTER last_known_addr { host: "127.1.170.190" port: 38203 } }
I20260923 20:14:39.334551 23655 leader_election.cc:302] T 00000000000000000000000000000000 P 49ab34c3f207402cbd179b60f0497365 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 49ab34c3f207402cbd179b60f0497365; no voters:
I20260923 20:14:39.335165 23655 leader_election.cc:288] T 00000000000000000000000000000000 P 49ab34c3f207402cbd179b60f0497365 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 20:14:39.335341 23665 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 49ab34c3f207402cbd179b60f0497365 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 20:14:39.336544 23665 raft_consensus.cc:686] T 00000000000000000000000000000000 P 49ab34c3f207402cbd179b60f0497365 [term 1 LEADER]: Becoming Leader. State: Replica: 49ab34c3f207402cbd179b60f0497365, State: Running, Role: LEADER
I20260923 20:14:39.337155 23665 consensus_queue.cc:237] T 00000000000000000000000000000000 P 49ab34c3f207402cbd179b60f0497365 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49ab34c3f207402cbd179b60f0497365" member_type: VOTER last_known_addr { host: "127.1.170.190" port: 38203 } }
I20260923 20:14:39.337689 23655 sys_catalog.cc:564] T 00000000000000000000000000000000 P 49ab34c3f207402cbd179b60f0497365 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 20:14:39.340211 23667 sys_catalog.cc:455] T 00000000000000000000000000000000 P 49ab34c3f207402cbd179b60f0497365 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 49ab34c3f207402cbd179b60f0497365. Latest consensus state: current_term: 1 leader_uuid: "49ab34c3f207402cbd179b60f0497365" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49ab34c3f207402cbd179b60f0497365" member_type: VOTER last_known_addr { host: "127.1.170.190" port: 38203 } } }
I20260923 20:14:39.340598 23667 sys_catalog.cc:458] T 00000000000000000000000000000000 P 49ab34c3f207402cbd179b60f0497365 [sys.catalog]: This master's current role is: LEADER
I20260923 20:14:39.341089 23666 sys_catalog.cc:455] T 00000000000000000000000000000000 P 49ab34c3f207402cbd179b60f0497365 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "49ab34c3f207402cbd179b60f0497365" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49ab34c3f207402cbd179b60f0497365" member_type: VOTER last_known_addr { host: "127.1.170.190" port: 38203 } } }
I20260923 20:14:39.341418 23666 sys_catalog.cc:458] T 00000000000000000000000000000000 P 49ab34c3f207402cbd179b60f0497365 [sys.catalog]: This master's current role is: LEADER
I20260923 20:14:39.341674 23673 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 20:14:39.349145 23673 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 20:14:39.360459 23673 catalog_manager.cc:1347] Generated new cluster ID: 615dd83161404229afa9b632e34929ff
I20260923 20:14:39.360599 23673 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 20:14:39.373106 23673 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 20:14:39.375082 23673 catalog_manager.cc:1504] Loading token signing keys...
I20260923 20:14:39.398663 23673 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 49ab34c3f207402cbd179b60f0497365: Generated new TSK 0
I20260923 20:14:39.399598 23673 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 20:14:39.415472 1706 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskqUuku6/build/asan/bin/kudu pid 23588
W20260923 20:14:39.417022 23650 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:41309
W20260923 20:14:39.418234 1706 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskqUuku6/build/asan/bin/kudu pid 23588: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260923 20:14:39.418469 1706 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqUuku6/build/asan/bin/kudu with pid 23588
W20260923 20:14:39.437139 23573 connection.cc:535] client connection to 127.1.170.190:38203 recv error: Network error: failed to read from TLS socket (remote: 127.1.170.190:38203): Connection reset by peer (error 104)
2026-09-23T20:14:39Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1989 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790194121577027-1706-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 20:14:39 dist-test-slave-th4g krb5kdc[23686](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 20:14:39 dist-test-slave-th4g krb5kdc[23686](info): set up 2 sockets
Sep 23 20:14:39 dist-test-slave-th4g krb5kdc[23686](info): commencing operation
krb5kdc: starting...
W20260923 20:14:40.562530 1706 mini_kdc.cc:121] Time spent starting KDC: real 1.105s user 0.000s sys 0.031s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790194121577027-1706-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790194121577027-1706-0/krb5kdc/test-user.keytab.
Sep 23 20:14:40 dist-test-slave-th4g krb5kdc[23686](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194480, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T20:14:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T20:14:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.170.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.170.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.170.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790194121577027-1706-0/krb5kdc/kudu_127.1.170.190.keytab.
Entry for principal kudu/127.1.170.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790194121577027-1706-0/krb5kdc/kudu_127.1.170.190.keytab.
I20260923 20:14:40.823100 1706 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqUuku6/build/asan/bin/kudu
/tmp/dist-test-taskqUuku6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790194121577027-1706-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790194121577027-1706-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790194121577027-1706-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790194121577027-1706-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.170.190:37539
--webserver_interface=127.1.170.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790194121577027-1706-0/krb5kdc/kudu_127.1.170.190.keytab
--principal=kudu/127.1.170.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.170.148:33227
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.170.190:37539
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790194121577027-1706-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790194121577027-1706-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790194121577027-1706-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:14:41.154150 23699 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:14:41.154422 23699 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:14:41.154608 23699 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:14:41.161993 23699 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 20:14:41.162124 23699 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:14:41.162230 23699 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:14:41.162335 23699 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 20:14:41.162408 23699 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 20:14:41.172150 23699 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.170.148:33227
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790194121577027-1706-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790194121577027-1706-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.170.190:37539
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790194121577027-1706-0/krb5kdc/kudu_127.1.170.190.keytab
--principal=kudu/127.1.170.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.170.190:37539
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790194121577027-1706-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.170.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790194121577027-1706-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:14:41.173498 23699 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:14:41.174815 23699 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:14:41.186834 23705 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:14:41.186884 23707 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:14:41.186914 23704 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:14:41.187965 23699 server_base.cc:1034] running on GCE node
I20260923 20:14:41.188771 23699 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:14:41.190266 23699 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:14:41.191512 23699 hybrid_clock.cc:648] HybridClock initialized: now 1790194481191465 us; error 38 us; skew 500 ppm
I20260923 20:14:41.191874 23699 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:14:41 dist-test-slave-th4g krb5kdc[23686](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194481, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.170.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:14:41.195073 23699 init.cc:377] Logged in from keytab as kudu/127.1.170.190@KRBTEST.COM (short username kudu)
I20260923 20:14:41.196545 23699 webserver.cc:460] Webserver started at http://127.1.170.190:33045/ using document root <none> and password file <none>
I20260923 20:14:41.197487 23699 fs_manager.cc:359] Metadata directory not provided
I20260923 20:14:41.197643 23699 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:14:41.198031 23699 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:14:41.200943 23699 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790194121577027-1706-0/minicluster-data/master-0/data/instance:
uuid: "231dd468ef444a51a8a589e9fa4e72db"
format_stamp: "Formatted at 2026-09-23 20:14:41 on dist-test-slave-th4g"
server_key: "293cc099c315af0253255e4bb0f28d89"
server_key_iv: "5ceb725763818b34c4cd0da5ce890daf"
server_key_version: "encryptionkey@0"
I20260923 20:14:41.201701 23699 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790194121577027-1706-0/minicluster-data/master-0/wal/instance:
uuid: "231dd468ef444a51a8a589e9fa4e72db"
format_stamp: "Formatted at 2026-09-23 20:14:41 on dist-test-slave-th4g"
server_key: "293cc099c315af0253255e4bb0f28d89"
server_key_iv: "5ceb725763818b34c4cd0da5ce890daf"
server_key_version: "encryptionkey@0"
I20260923 20:14:41.206790 23699 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260923 20:14:41.210260 23714 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:14:41.211562 23699 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260923 20:14:41.211746 23699 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790194121577027-1706-0/minicluster-data/master-0/data,/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790194121577027-1706-0/minicluster-data/master-0/wal
uuid: "231dd468ef444a51a8a589e9fa4e72db"
format_stamp: "Formatted at 2026-09-23 20:14:41 on dist-test-slave-th4g"
server_key: "293cc099c315af0253255e4bb0f28d89"
server_key_iv: "5ceb725763818b34c4cd0da5ce890daf"
server_key_version: "encryptionkey@0"
I20260923 20:14:41.211966 23699 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790194121577027-1706-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790194121577027-1706-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790194121577027-1706-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:14:41.218014 23699 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:14:41.222077 23699 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:14:41.222374 23699 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:14:41.241487 23699 rpc_server.cc:307] RPC server started. Bound to: 127.1.170.190:37539
I20260923 20:14:41.241523 23765 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.170.190:37539 every 8 connection(s)
I20260923 20:14:41.242887 23699 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790194121577027-1706-0/minicluster-data/master-0/data/info.pb
I20260923 20:14:41.246766 23766 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:14:41.247503 1706 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqUuku6/build/asan/bin/kudu as pid 23699
I20260923 20:14:41.247691 1706 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790194121577027-1706-0/minicluster-data/master-0/wal/instance
I20260923 20:14:41.248245 1706 external_mini_cluster.cc:1428] Setting key 0316eab3e93f8528790f74619ad8a7a3
I20260923 20:14:41.256558 23769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:14:41.250196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33079 (local address 127.1.170.190:37539)
0923 20:14:41.251001 (+ 805us) server_negotiation.cc:206] Beginning negotiation
0923 20:14:41.251017 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0923 20:14:41.251057 (+ 40us) server_negotiation.cc:407] Connection header received
0923 20:14:41.253209 (+ 2152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:14:41.253230 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:14:41.253597 (+ 367us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:14:41.254027 (+ 430us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:14:41.255040 (+ 1013us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:14:41.255047 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:14:41.255059 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 20:14:41.255072 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:14:41.255127 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 20:14:41.255150 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:14:41.255316 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0923 20:14:41.255413 (+ 97us) server_negotiation.cc:299] Negotiation successful
0923 20:14:41.255597 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":567,"thread_start_us":361,"threads_started":1}
I20260923 20:14:41.260284 1706 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 20:14:41.260392 23766 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 231dd468ef444a51a8a589e9fa4e72db: Bootstrap starting.
I20260923 20:14:41.264186 23766 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 231dd468ef444a51a8a589e9fa4e72db: Neither blocks nor log segments found. Creating new log.
I20260923 20:14:41.265553 23766 log.cc:824] T 00000000000000000000000000000000 P 231dd468ef444a51a8a589e9fa4e72db: Log is configured to *not* fsync() on all Append() calls
I20260923 20:14:41.268137 23769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:14:41.265465 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48943 (local address 127.1.170.190:37539)
0923 20:14:41.265656 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0923 20:14:41.265663 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:14:41.266150 (+ 487us) server_negotiation.cc:407] Connection header received
0923 20:14:41.266347 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:14:41.266351 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:14:41.266396 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:14:41.266478 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:14:41.267511 (+ 1033us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:14:41.267516 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:14:41.267521 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 20:14:41.267539 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:14:41.267582 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 20:14:41.267601 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:14:41.267717 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0923 20:14:41.267794 (+ 77us) server_negotiation.cc:299] Negotiation successful
0923 20:14:41.267861 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260923 20:14:41.268819 23766 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 231dd468ef444a51a8a589e9fa4e72db: No bootstrap required, opened a new log
I20260923 20:14:41.275914 23766 raft_consensus.cc:348] T 00000000000000000000000000000000 P 231dd468ef444a51a8a589e9fa4e72db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "231dd468ef444a51a8a589e9fa4e72db" member_type: VOTER last_known_addr { host: "127.1.170.190" port: 37539 } }
I20260923 20:14:41.276263 23766 raft_consensus.cc:374] T 00000000000000000000000000000000 P 231dd468ef444a51a8a589e9fa4e72db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:14:41.276422 23766 raft_consensus.cc:729] T 00000000000000000000000000000000 P 231dd468ef444a51a8a589e9fa4e72db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 231dd468ef444a51a8a589e9fa4e72db, State: Initialized, Role: FOLLOWER
I20260923 20:14:41.276983 23766 consensus_queue.cc:260] T 00000000000000000000000000000000 P 231dd468ef444a51a8a589e9fa4e72db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "231dd468ef444a51a8a589e9fa4e72db" member_type: VOTER last_known_addr { host: "127.1.170.190" port: 37539 } }
I20260923 20:14:41.277267 23766 raft_consensus.cc:388] T 00000000000000000000000000000000 P 231dd468ef444a51a8a589e9fa4e72db [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 20:14:41.277423 23766 raft_consensus.cc:482] T 00000000000000000000000000000000 P 231dd468ef444a51a8a589e9fa4e72db [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 20:14:41.277609 23766 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 231dd468ef444a51a8a589e9fa4e72db [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:14:41.279809 23766 raft_consensus.cc:504] T 00000000000000000000000000000000 P 231dd468ef444a51a8a589e9fa4e72db [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "231dd468ef444a51a8a589e9fa4e72db" member_type: VOTER last_known_addr { host: "127.1.170.190" port: 37539 } }
I20260923 20:14:41.280324 23766 leader_election.cc:302] T 00000000000000000000000000000000 P 231dd468ef444a51a8a589e9fa4e72db [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 231dd468ef444a51a8a589e9fa4e72db; no voters:
I20260923 20:14:41.280965 23766 leader_election.cc:288] T 00000000000000000000000000000000 P 231dd468ef444a51a8a589e9fa4e72db [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 20:14:41.281149 23776 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 231dd468ef444a51a8a589e9fa4e72db [term 1 FOLLOWER]: Leader election won for term 1
I20260923 20:14:41.282246 23776 raft_consensus.cc:686] T 00000000000000000000000000000000 P 231dd468ef444a51a8a589e9fa4e72db [term 1 LEADER]: Becoming Leader. State: Replica: 231dd468ef444a51a8a589e9fa4e72db, State: Running, Role: LEADER
I20260923 20:14:41.282757 23776 consensus_queue.cc:237] T 00000000000000000000000000000000 P 231dd468ef444a51a8a589e9fa4e72db [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "231dd468ef444a51a8a589e9fa4e72db" member_type: VOTER last_known_addr { host: "127.1.170.190" port: 37539 } }
I20260923 20:14:41.283283 23766 sys_catalog.cc:564] T 00000000000000000000000000000000 P 231dd468ef444a51a8a589e9fa4e72db [sys.catalog]: configured and running, proceeding with master startup.
I20260923 20:14:41.285993 23778 sys_catalog.cc:455] T 00000000000000000000000000000000 P 231dd468ef444a51a8a589e9fa4e72db [sys.catalog]: SysCatalogTable state changed. Reason: New leader 231dd468ef444a51a8a589e9fa4e72db. Latest consensus state: current_term: 1 leader_uuid: "231dd468ef444a51a8a589e9fa4e72db" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "231dd468ef444a51a8a589e9fa4e72db" member_type: VOTER last_known_addr { host: "127.1.170.190" port: 37539 } } }
I20260923 20:14:41.286561 23778 sys_catalog.cc:458] T 00000000000000000000000000000000 P 231dd468ef444a51a8a589e9fa4e72db [sys.catalog]: This master's current role is: LEADER
I20260923 20:14:41.286480 23777 sys_catalog.cc:455] T 00000000000000000000000000000000 P 231dd468ef444a51a8a589e9fa4e72db [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "231dd468ef444a51a8a589e9fa4e72db" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "231dd468ef444a51a8a589e9fa4e72db" member_type: VOTER last_known_addr { host: "127.1.170.190" port: 37539 } } }
I20260923 20:14:41.287039 23777 sys_catalog.cc:458] T 00000000000000000000000000000000 P 231dd468ef444a51a8a589e9fa4e72db [sys.catalog]: This master's current role is: LEADER
I20260923 20:14:41.294832 23785 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 20:14:41.300276 23785 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 20:14:41.309643 23785 catalog_manager.cc:1347] Generated new cluster ID: 2580bece15be4906be999c5d92630bd3
I20260923 20:14:41.309785 23785 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 20:14:41.320294 23785 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 20:14:41.320749 23785 catalog_manager.cc:1504] Loading token signing keys...
I20260923 20:14:41.334671 23785 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 231dd468ef444a51a8a589e9fa4e72db: Generated new TSK 0
I20260923 20:14:41.335590 23785 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 20:14:41.343858 1706 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskqUuku6/build/asan/bin/kudu pid 23699
W20260923 20:14:41.344997 23761 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:33079
W20260923 20:14:41.345726 1706 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskqUuku6/build/asan/bin/kudu pid 23699: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260923 20:14:41.345916 1706 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqUuku6/build/asan/bin/kudu with pid 23699
2026-09-23T20:14:41Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1925 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790194121577027-1706-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 20:14:41 dist-test-slave-th4g krb5kdc[23797](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 20:14:41 dist-test-slave-th4g krb5kdc[23797](info): set up 2 sockets
Sep 23 20:14:41 dist-test-slave-th4g krb5kdc[23797](info): commencing operation
krb5kdc: starting...
W20260923 20:14:42.463147 1706 mini_kdc.cc:121] Time spent starting KDC: real 1.081s user 0.000s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790194121577027-1706-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790194121577027-1706-0/krb5kdc/test-user.keytab.
Sep 23 20:14:42 dist-test-slave-th4g krb5kdc[23797](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194482, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T20:14:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T20:14:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.170.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.170.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.170.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790194121577027-1706-0/krb5kdc/kudu_127.1.170.190.keytab.
Entry for principal kudu/127.1.170.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790194121577027-1706-0/krb5kdc/kudu_127.1.170.190.keytab.
I20260923 20:14:42.738898 1706 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqUuku6/build/asan/bin/kudu
/tmp/dist-test-taskqUuku6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790194121577027-1706-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790194121577027-1706-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790194121577027-1706-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790194121577027-1706-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.170.190:40945
--webserver_interface=127.1.170.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790194121577027-1706-0/krb5kdc/kudu_127.1.170.190.keytab
--principal=kudu/127.1.170.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.170.148:41801
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.170.190:40945
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790194121577027-1706-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790194121577027-1706-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790194121577027-1706-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:14:43.085316 23810 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:14:43.085614 23810 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:14:43.085799 23810 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:14:43.094154 23810 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 20:14:43.094347 23810 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:14:43.094439 23810 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:14:43.094518 23810 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 20:14:43.094558 23810 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 20:14:43.105177 23810 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.170.148:41801
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790194121577027-1706-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790194121577027-1706-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.170.190:40945
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790194121577027-1706-0/krb5kdc/kudu_127.1.170.190.keytab
--principal=kudu/127.1.170.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.170.190:40945
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790194121577027-1706-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.170.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790194121577027-1706-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:14:43.106550 23810 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:14:43.107810 23810 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:14:43.120860 23815 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:14:43.120945 23816 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:14:43.120795 23818 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:14:43.122048 23810 server_base.cc:1034] running on GCE node
I20260923 20:14:43.122882 23810 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:14:43.124446 23810 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:14:43.125682 23810 hybrid_clock.cc:648] HybridClock initialized: now 1790194483125622 us; error 56 us; skew 500 ppm
I20260923 20:14:43.126155 23810 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:14:43 dist-test-slave-th4g krb5kdc[23797](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194483, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.170.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:14:43.135610 23810 init.cc:377] Logged in from keytab as kudu/127.1.170.190@KRBTEST.COM (short username kudu)
I20260923 20:14:43.137504 23810 webserver.cc:460] Webserver started at http://127.1.170.190:45835/ using document root <none> and password file <none>
I20260923 20:14:43.138389 23810 fs_manager.cc:359] Metadata directory not provided
I20260923 20:14:43.138540 23810 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:14:43.138906 23810 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:14:43.142413 23810 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790194121577027-1706-0/minicluster-data/master-0/data/instance:
uuid: "f71950133ba54f79984b576be1215890"
format_stamp: "Formatted at 2026-09-23 20:14:43 on dist-test-slave-th4g"
server_key: "9f6db6f1d837ba2568dba0a66d826df1"
server_key_iv: "5d765b0544a8cc211e3e1218f3b2d3cb"
server_key_version: "encryptionkey@0"
I20260923 20:14:43.143317 23810 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790194121577027-1706-0/minicluster-data/master-0/wal/instance:
uuid: "f71950133ba54f79984b576be1215890"
format_stamp: "Formatted at 2026-09-23 20:14:43 on dist-test-slave-th4g"
server_key: "9f6db6f1d837ba2568dba0a66d826df1"
server_key_iv: "5d765b0544a8cc211e3e1218f3b2d3cb"
server_key_version: "encryptionkey@0"
I20260923 20:14:43.149281 23810 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260923 20:14:43.153215 23825 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:14:43.154775 23810 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 20:14:43.154976 23810 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790194121577027-1706-0/minicluster-data/master-0/data,/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790194121577027-1706-0/minicluster-data/master-0/wal
uuid: "f71950133ba54f79984b576be1215890"
format_stamp: "Formatted at 2026-09-23 20:14:43 on dist-test-slave-th4g"
server_key: "9f6db6f1d837ba2568dba0a66d826df1"
server_key_iv: "5d765b0544a8cc211e3e1218f3b2d3cb"
server_key_version: "encryptionkey@0"
I20260923 20:14:43.155192 23810 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790194121577027-1706-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790194121577027-1706-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790194121577027-1706-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:14:43.176788 23810 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:14:43.181200 23810 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:14:43.181528 23810 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:14:43.201711 23810 rpc_server.cc:307] RPC server started. Bound to: 127.1.170.190:40945
I20260923 20:14:43.201718 23876 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.170.190:40945 every 8 connection(s)
I20260923 20:14:43.203220 23810 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790194121577027-1706-0/minicluster-data/master-0/data/info.pb
I20260923 20:14:43.206811 1706 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqUuku6/build/asan/bin/kudu as pid 23810
I20260923 20:14:43.207026 1706 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790194121577027-1706-0/minicluster-data/master-0/wal/instance
I20260923 20:14:43.207157 23877 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:14:43.207536 1706 external_mini_cluster.cc:1428] Setting key b5479cdbf21d900f42f18a8c47a847db
I20260923 20:14:43.222375 23877 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f71950133ba54f79984b576be1215890: Bootstrap starting.
Sep 23 20:14:43 dist-test-slave-th4g krb5kdc[23797](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194482, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.170.190@KRBTEST.COM
I20260923 20:14:43.226383 23877 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f71950133ba54f79984b576be1215890: Neither blocks nor log segments found. Creating new log.
I20260923 20:14:43.227811 23877 log.cc:824] T 00000000000000000000000000000000 P f71950133ba54f79984b576be1215890: Log is configured to *not* fsync() on all Append() calls
I20260923 20:14:43.231570 23877 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f71950133ba54f79984b576be1215890: No bootstrap required, opened a new log
I20260923 20:14:43.233120 23880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:14:43.209276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52275 (local address 127.1.170.190:40945)
0923 20:14:43.210141 (+ 865us) server_negotiation.cc:206] Beginning negotiation
0923 20:14:43.210153 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0923 20:14:43.210209 (+ 56us) server_negotiation.cc:407] Connection header received
0923 20:14:43.212038 (+ 1829us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:14:43.212063 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:14:43.212515 (+ 452us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:14:43.212877 (+ 362us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:14:43.214779 (+ 1902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:14:43.216147 (+ 1368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:14:43.218012 (+ 1865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:14:43.218715 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:14:43.218869 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:14:43.223873 (+ 5004us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:14:43.223887 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:14:43.223912 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:14:43.223975 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:14:43.228172 (+ 4197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:14:43.229183 (+ 1011us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:14:43.229191 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:14:43.229200 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:14:43.229454 (+ 254us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:14:43.229928 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:14:43.229934 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:14:43.229937 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:14:43.230646 (+ 709us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:14:43.230953 (+ 307us) server_negotiation.cc:1035] Waiting for connection context
0923 20:14:43.231339 (+ 386us) server_negotiation.cc:299] Negotiation successful
0923 20:14:43.231764 (+ 425us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":656,"thread_start_us":396,"threads_started":1}
I20260923 20:14:43.237145 1706 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 20:14:43.240525 23877 raft_consensus.cc:348] T 00000000000000000000000000000000 P f71950133ba54f79984b576be1215890 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f71950133ba54f79984b576be1215890" member_type: VOTER last_known_addr { host: "127.1.170.190" port: 40945 } }
I20260923 20:14:43.240937 23877 raft_consensus.cc:374] T 00000000000000000000000000000000 P f71950133ba54f79984b576be1215890 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:14:43.241050 23877 raft_consensus.cc:729] T 00000000000000000000000000000000 P f71950133ba54f79984b576be1215890 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f71950133ba54f79984b576be1215890, State: Initialized, Role: FOLLOWER
I20260923 20:14:43.241746 23877 consensus_queue.cc:260] T 00000000000000000000000000000000 P f71950133ba54f79984b576be1215890 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f71950133ba54f79984b576be1215890" member_type: VOTER last_known_addr { host: "127.1.170.190" port: 40945 } }
I20260923 20:14:43.242096 23877 raft_consensus.cc:388] T 00000000000000000000000000000000 P f71950133ba54f79984b576be1215890 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 20:14:43.242262 23877 raft_consensus.cc:482] T 00000000000000000000000000000000 P f71950133ba54f79984b576be1215890 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 20:14:43.242452 23877 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f71950133ba54f79984b576be1215890 [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:14:43.245730 23877 raft_consensus.cc:504] T 00000000000000000000000000000000 P f71950133ba54f79984b576be1215890 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f71950133ba54f79984b576be1215890" member_type: VOTER last_known_addr { host: "127.1.170.190" port: 40945 } }
I20260923 20:14:43.246438 23877 leader_election.cc:302] T 00000000000000000000000000000000 P f71950133ba54f79984b576be1215890 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f71950133ba54f79984b576be1215890; no voters:
I20260923 20:14:43.247162 23877 leader_election.cc:288] T 00000000000000000000000000000000 P f71950133ba54f79984b576be1215890 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 20:14:43.247287 23887 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f71950133ba54f79984b576be1215890 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 20:14:43.248620 23887 raft_consensus.cc:686] T 00000000000000000000000000000000 P f71950133ba54f79984b576be1215890 [term 1 LEADER]: Becoming Leader. State: Replica: f71950133ba54f79984b576be1215890, State: Running, Role: LEADER
I20260923 20:14:43.249620 23887 consensus_queue.cc:237] T 00000000000000000000000000000000 P f71950133ba54f79984b576be1215890 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f71950133ba54f79984b576be1215890" member_type: VOTER last_known_addr { host: "127.1.170.190" port: 40945 } }
I20260923 20:14:43.250562 23877 sys_catalog.cc:564] T 00000000000000000000000000000000 P f71950133ba54f79984b576be1215890 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 20:14:43.253495 23889 sys_catalog.cc:455] T 00000000000000000000000000000000 P f71950133ba54f79984b576be1215890 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f71950133ba54f79984b576be1215890. Latest consensus state: current_term: 1 leader_uuid: "f71950133ba54f79984b576be1215890" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f71950133ba54f79984b576be1215890" member_type: VOTER last_known_addr { host: "127.1.170.190" port: 40945 } } }
I20260923 20:14:43.253492 23888 sys_catalog.cc:455] T 00000000000000000000000000000000 P f71950133ba54f79984b576be1215890 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f71950133ba54f79984b576be1215890" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f71950133ba54f79984b576be1215890" member_type: VOTER last_known_addr { host: "127.1.170.190" port: 40945 } } }
I20260923 20:14:43.255041 23889 sys_catalog.cc:458] T 00000000000000000000000000000000 P f71950133ba54f79984b576be1215890 [sys.catalog]: This master's current role is: LEADER
I20260923 20:14:43.255082 23888 sys_catalog.cc:458] T 00000000000000000000000000000000 P f71950133ba54f79984b576be1215890 [sys.catalog]: This master's current role is: LEADER
I20260923 20:14:43.256575 23892 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 20:14:43.261018 23880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:14:43.241574 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34359 (local address 127.1.170.190:40945)
0923 20:14:43.241780 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0923 20:14:43.241786 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:14:43.242268 (+ 482us) server_negotiation.cc:407] Connection header received
0923 20:14:43.242470 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:14:43.242476 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:14:43.242732 (+ 256us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:14:43.243122 (+ 390us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:14:43.244536 (+ 1414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:14:43.245628 (+ 1092us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:14:43.247174 (+ 1546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:14:43.248043 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:14:43.248181 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:14:43.251095 (+ 2914us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:14:43.251101 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:14:43.251104 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:14:43.251141 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:14:43.256228 (+ 5087us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:14:43.257595 (+ 1367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:14:43.257606 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:14:43.257609 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:14:43.257766 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:14:43.258278 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:14:43.258285 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:14:43.258292 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:14:43.258845 (+ 553us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:14:43.260229 (+ 1384us) server_negotiation.cc:1035] Waiting for connection context
0923 20:14:43.260391 (+ 162us) server_negotiation.cc:299] Negotiation successful
0923 20:14:43.260632 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260923 20:14:43.262105 23892 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 20:14:43.276959 23892 catalog_manager.cc:1347] Generated new cluster ID: 52c39449a2294a99bb87123583b32b81
I20260923 20:14:43.277107 23892 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 20:14:43.297627 23892 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 20:14:43.299098 23892 catalog_manager.cc:1504] Loading token signing keys...
I20260923 20:14:43.307149 23892 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f71950133ba54f79984b576be1215890: Generated new TSK 0
I20260923 20:14:43.308012 23892 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 20:14:43.316709 1706 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskqUuku6/build/asan/bin/kudu pid 23810
I20260923 20:14:43.318063 23872 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:14:43.411378 1706 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqUuku6/build/asan/bin/kudu with pid 23810
2026-09-23T20:14:43Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (2064 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790194121577027-1706-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 20:14:43 dist-test-slave-th4g krb5kdc[23909](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 20:14:43 dist-test-slave-th4g krb5kdc[23909](info): set up 2 sockets
Sep 23 20:14:43 dist-test-slave-th4g krb5kdc[23909](info): commencing operation
krb5kdc: starting...
W20260923 20:14:44.542598 1706 mini_kdc.cc:121] Time spent starting KDC: real 1.096s user 0.000s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790194121577027-1706-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790194121577027-1706-0/krb5kdc/test-user.keytab.
Sep 23 20:14:44 dist-test-slave-th4g krb5kdc[23909](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194484, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T20:14:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T20:14:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.170.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.170.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.170.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790194121577027-1706-0/krb5kdc/kudu_127.1.170.190.keytab.
Entry for principal kudu/127.1.170.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790194121577027-1706-0/krb5kdc/kudu_127.1.170.190.keytab.
I20260923 20:14:44.806182 1706 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqUuku6/build/asan/bin/kudu
/tmp/dist-test-taskqUuku6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790194121577027-1706-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790194121577027-1706-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790194121577027-1706-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790194121577027-1706-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.170.190:33563
--webserver_interface=127.1.170.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790194121577027-1706-0/krb5kdc/kudu_127.1.170.190.keytab
--principal=kudu/127.1.170.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.170.148:42561
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.170.190:33563
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790194121577027-1706-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790194121577027-1706-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790194121577027-1706-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:14:45.135124 23922 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:14:45.135458 23922 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:14:45.135558 23922 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:14:45.142674 23922 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 20:14:45.142786 23922 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:14:45.142840 23922 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:14:45.142894 23922 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 20:14:45.142939 23922 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 20:14:45.152943 23922 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.170.148:42561
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790194121577027-1706-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790194121577027-1706-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.170.190:33563
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790194121577027-1706-0/krb5kdc/kudu_127.1.170.190.keytab
--principal=kudu/127.1.170.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.170.190:33563
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790194121577027-1706-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.170.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790194121577027-1706-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:14:45.154320 23922 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:14:45.155608 23922 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:14:45.166790 23927 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:14:45.166954 23928 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:14:45.166993 23930 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:14:45.167992 23922 server_base.cc:1034] running on GCE node
I20260923 20:14:45.168689 23922 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:14:45.170061 23922 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:14:45.171272 23922 hybrid_clock.cc:648] HybridClock initialized: now 1790194485171222 us; error 41 us; skew 500 ppm
I20260923 20:14:45.171569 23922 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:14:45 dist-test-slave-th4g krb5kdc[23909](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194485, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.170.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:14:45.174701 23922 init.cc:377] Logged in from keytab as kudu/127.1.170.190@KRBTEST.COM (short username kudu)
I20260923 20:14:45.176288 23922 webserver.cc:460] Webserver started at http://127.1.170.190:33571/ using document root <none> and password file <none>
I20260923 20:14:45.176976 23922 fs_manager.cc:359] Metadata directory not provided
I20260923 20:14:45.177122 23922 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:14:45.177415 23922 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:14:45.180380 23922 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790194121577027-1706-0/minicluster-data/master-0/data/instance:
uuid: "40b38af0fe0d4d68969724c803a1595e"
format_stamp: "Formatted at 2026-09-23 20:14:45 on dist-test-slave-th4g"
server_key: "c4681435a5f00072d9a27a590c08ebcb"
server_key_iv: "abdff781cb64406d9bad316e454ab3d4"
server_key_version: "encryptionkey@0"
I20260923 20:14:45.181144 23922 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790194121577027-1706-0/minicluster-data/master-0/wal/instance:
uuid: "40b38af0fe0d4d68969724c803a1595e"
format_stamp: "Formatted at 2026-09-23 20:14:45 on dist-test-slave-th4g"
server_key: "c4681435a5f00072d9a27a590c08ebcb"
server_key_iv: "abdff781cb64406d9bad316e454ab3d4"
server_key_version: "encryptionkey@0"
I20260923 20:14:45.186307 23922 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260923 20:14:45.189719 23937 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:14:45.191097 23922 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260923 20:14:45.191238 23922 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790194121577027-1706-0/minicluster-data/master-0/data,/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790194121577027-1706-0/minicluster-data/master-0/wal
uuid: "40b38af0fe0d4d68969724c803a1595e"
format_stamp: "Formatted at 2026-09-23 20:14:45 on dist-test-slave-th4g"
server_key: "c4681435a5f00072d9a27a590c08ebcb"
server_key_iv: "abdff781cb64406d9bad316e454ab3d4"
server_key_version: "encryptionkey@0"
I20260923 20:14:45.191399 23922 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790194121577027-1706-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790194121577027-1706-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790194121577027-1706-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:14:45.216648 23922 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:14:45.220944 23922 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:14:45.221176 23922 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:14:45.240216 23922 rpc_server.cc:307] RPC server started. Bound to: 127.1.170.190:33563
I20260923 20:14:45.240214 23988 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.170.190:33563 every 8 connection(s)
I20260923 20:14:45.241592 23922 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790194121577027-1706-0/minicluster-data/master-0/data/info.pb
I20260923 20:14:45.245236 23989 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:14:45.250629 1706 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqUuku6/build/asan/bin/kudu as pid 23922
I20260923 20:14:45.250823 1706 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790194121577027-1706-0/minicluster-data/master-0/wal/instance
I20260923 20:14:45.251276 1706 external_mini_cluster.cc:1428] Setting key ee423e1f8fda2a58f38850732622c1e1
I20260923 20:14:45.255983 23989 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 40b38af0fe0d4d68969724c803a1595e: Bootstrap starting.
I20260923 20:14:45.259536 23989 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 40b38af0fe0d4d68969724c803a1595e: Neither blocks nor log segments found. Creating new log.
I20260923 20:14:45.260766 23989 log.cc:824] T 00000000000000000000000000000000 P 40b38af0fe0d4d68969724c803a1595e: Log is configured to *not* fsync() on all Append() calls
I20260923 20:14:45.263834 23989 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 40b38af0fe0d4d68969724c803a1595e: No bootstrap required, opened a new log
I20260923 20:14:45.264835 23992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:14:45.253149 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46835 (local address 127.1.170.190:33563)
0923 20:14:45.254032 (+ 883us) server_negotiation.cc:206] Beginning negotiation
0923 20:14:45.254044 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0923 20:14:45.254090 (+ 46us) server_negotiation.cc:407] Connection header received
0923 20:14:45.255754 (+ 1664us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:14:45.255774 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:14:45.256202 (+ 428us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:14:45.256531 (+ 329us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:14:45.257120 (+ 589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:14:45.258514 (+ 1394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:14:45.260359 (+ 1845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:14:45.261086 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:14:45.261256 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:14:45.262536 (+ 1280us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:14:45.262543 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:14:45.262555 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 20:14:45.263082 (+ 527us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:14:45.263148 (+ 66us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 20:14:45.263176 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:14:45.263265 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0923 20:14:45.263491 (+ 226us) server_negotiation.cc:299] Negotiation successful
0923 20:14:45.263788 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":631,"thread_start_us":398,"threads_started":1}
I20260923 20:14:45.268330 1706 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 20:14:45.271971 23989 raft_consensus.cc:348] T 00000000000000000000000000000000 P 40b38af0fe0d4d68969724c803a1595e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40b38af0fe0d4d68969724c803a1595e" member_type: VOTER last_known_addr { host: "127.1.170.190" port: 33563 } }
I20260923 20:14:45.272426 23989 raft_consensus.cc:374] T 00000000000000000000000000000000 P 40b38af0fe0d4d68969724c803a1595e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:14:45.272562 23989 raft_consensus.cc:729] T 00000000000000000000000000000000 P 40b38af0fe0d4d68969724c803a1595e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 40b38af0fe0d4d68969724c803a1595e, State: Initialized, Role: FOLLOWER
I20260923 20:14:45.273430 23989 consensus_queue.cc:260] T 00000000000000000000000000000000 P 40b38af0fe0d4d68969724c803a1595e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40b38af0fe0d4d68969724c803a1595e" member_type: VOTER last_known_addr { host: "127.1.170.190" port: 33563 } }
I20260923 20:14:45.273790 23989 raft_consensus.cc:388] T 00000000000000000000000000000000 P 40b38af0fe0d4d68969724c803a1595e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 20:14:45.274009 23989 raft_consensus.cc:482] T 00000000000000000000000000000000 P 40b38af0fe0d4d68969724c803a1595e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 20:14:45.274196 23989 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 40b38af0fe0d4d68969724c803a1595e [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:14:45.276463 23989 raft_consensus.cc:504] T 00000000000000000000000000000000 P 40b38af0fe0d4d68969724c803a1595e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40b38af0fe0d4d68969724c803a1595e" member_type: VOTER last_known_addr { host: "127.1.170.190" port: 33563 } }
I20260923 20:14:45.276952 23989 leader_election.cc:302] T 00000000000000000000000000000000 P 40b38af0fe0d4d68969724c803a1595e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 40b38af0fe0d4d68969724c803a1595e; no voters:
I20260923 20:14:45.277611 23989 leader_election.cc:288] T 00000000000000000000000000000000 P 40b38af0fe0d4d68969724c803a1595e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 20:14:45.277781 23999 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 40b38af0fe0d4d68969724c803a1595e [term 1 FOLLOWER]: Leader election won for term 1
I20260923 20:14:45.278642 23999 raft_consensus.cc:686] T 00000000000000000000000000000000 P 40b38af0fe0d4d68969724c803a1595e [term 1 LEADER]: Becoming Leader. State: Replica: 40b38af0fe0d4d68969724c803a1595e, State: Running, Role: LEADER
I20260923 20:14:45.279394 23999 consensus_queue.cc:237] T 00000000000000000000000000000000 P 40b38af0fe0d4d68969724c803a1595e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40b38af0fe0d4d68969724c803a1595e" member_type: VOTER last_known_addr { host: "127.1.170.190" port: 33563 } }
I20260923 20:14:45.280267 23992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:14:45.272696 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58189 (local address 127.1.170.190:33563)
0923 20:14:45.272886 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0923 20:14:45.272891 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 20:14:45.274457 (+ 1566us) server_negotiation.cc:407] Connection header received
0923 20:14:45.274674 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:14:45.274679 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:14:45.274749 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:14:45.274876 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:14:45.275355 (+ 479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:14:45.276055 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:14:45.277483 (+ 1428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:14:45.278116 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:14:45.278268 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:14:45.279503 (+ 1235us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:14:45.279507 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:14:45.279510 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 20:14:45.279526 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:14:45.279564 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 20:14:45.279582 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:14:45.279679 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0923 20:14:45.279832 (+ 153us) server_negotiation.cc:299] Negotiation successful
0923 20:14:45.279916 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260923 20:14:45.282859 24001 sys_catalog.cc:455] T 00000000000000000000000000000000 P 40b38af0fe0d4d68969724c803a1595e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "40b38af0fe0d4d68969724c803a1595e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40b38af0fe0d4d68969724c803a1595e" member_type: VOTER last_known_addr { host: "127.1.170.190" port: 33563 } } }
I20260923 20:14:45.283433 24001 sys_catalog.cc:458] T 00000000000000000000000000000000 P 40b38af0fe0d4d68969724c803a1595e [sys.catalog]: This master's current role is: LEADER
I20260923 20:14:45.283759 24003 sys_catalog.cc:455] T 00000000000000000000000000000000 P 40b38af0fe0d4d68969724c803a1595e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 40b38af0fe0d4d68969724c803a1595e. Latest consensus state: current_term: 1 leader_uuid: "40b38af0fe0d4d68969724c803a1595e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40b38af0fe0d4d68969724c803a1595e" member_type: VOTER last_known_addr { host: "127.1.170.190" port: 33563 } } }
I20260923 20:14:45.284547 24003 sys_catalog.cc:458] T 00000000000000000000000000000000 P 40b38af0fe0d4d68969724c803a1595e [sys.catalog]: This master's current role is: LEADER
I20260923 20:14:45.286262 23989 sys_catalog.cc:564] T 00000000000000000000000000000000 P 40b38af0fe0d4d68969724c803a1595e [sys.catalog]: configured and running, proceeding with master startup.
I20260923 20:14:45.286489 24004 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 20:14:45.292122 24004 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 20:14:45.302026 24004 catalog_manager.cc:1347] Generated new cluster ID: cdb2d569e1c446c4931185581ea0d5f0
I20260923 20:14:45.302196 24004 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 20:14:45.321465 24004 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 20:14:45.323513 24004 catalog_manager.cc:1504] Loading token signing keys...
I20260923 20:14:45.348111 24004 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 40b38af0fe0d4d68969724c803a1595e: Generated new TSK 0
I20260923 20:14:45.349087 24004 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 20:14:45.363117 1706 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskqUuku6/build/asan/bin/kudu pid 23922
W20260923 20:14:45.364223 23984 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:46835
W20260923 20:14:45.364965 1706 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskqUuku6/build/asan/bin/kudu pid 23922: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260923 20:14:45.365123 1706 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqUuku6/build/asan/bin/kudu with pid 23922
2026-09-23T20:14:45Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1952 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790194121577027-1706-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 20:14:45 dist-test-slave-th4g krb5kdc[24021](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 20:14:45 dist-test-slave-th4g krb5kdc[24021](info): set up 2 sockets
Sep 23 20:14:45 dist-test-slave-th4g krb5kdc[24021](info): commencing operation
krb5kdc: starting...
W20260923 20:14:46.498593 1706 mini_kdc.cc:121] Time spent starting KDC: real 1.099s user 0.001s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790194121577027-1706-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790194121577027-1706-0/krb5kdc/test-user.keytab.
Sep 23 20:14:46 dist-test-slave-th4g krb5kdc[24021](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194486, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T20:14:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T20:14:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.170.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.170.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.170.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790194121577027-1706-0/krb5kdc/kudu_127.1.170.190.keytab.
Entry for principal kudu/127.1.170.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790194121577027-1706-0/krb5kdc/kudu_127.1.170.190.keytab.
I20260923 20:14:46.766179 1706 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqUuku6/build/asan/bin/kudu
/tmp/dist-test-taskqUuku6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790194121577027-1706-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790194121577027-1706-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790194121577027-1706-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790194121577027-1706-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.170.190:33941
--webserver_interface=127.1.170.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790194121577027-1706-0/krb5kdc/kudu_127.1.170.190.keytab
--principal=kudu/127.1.170.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.170.148:43679
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.170.190:33941
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790194121577027-1706-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790194121577027-1706-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790194121577027-1706-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:14:47.114017 24034 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:14:47.114387 24034 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:14:47.114553 24034 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:14:47.122068 24034 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 20:14:47.122220 24034 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:14:47.122305 24034 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:14:47.122399 24034 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 20:14:47.122468 24034 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 20:14:47.138577 24034 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.170.148:43679
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790194121577027-1706-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790194121577027-1706-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.170.190:33941
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790194121577027-1706-0/krb5kdc/kudu_127.1.170.190.keytab
--principal=kudu/127.1.170.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.170.190:33941
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790194121577027-1706-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.170.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790194121577027-1706-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:14:47.140903 24034 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:14:47.142899 24034 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:14:47.158133 24039 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:14:47.158243 24040 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:14:47.158489 24042 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:14:47.159562 24034 server_base.cc:1034] running on GCE node
I20260923 20:14:47.160800 24034 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:14:47.162858 24034 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:14:47.164116 24034 hybrid_clock.cc:648] HybridClock initialized: now 1790194487164056 us; error 49 us; skew 500 ppm
I20260923 20:14:47.164549 24034 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:14:47 dist-test-slave-th4g krb5kdc[24021](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194487, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.170.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:14:47.169624 24034 init.cc:377] Logged in from keytab as kudu/127.1.170.190@KRBTEST.COM (short username kudu)
I20260923 20:14:47.171658 24034 webserver.cc:460] Webserver started at http://127.1.170.190:39197/ using document root <none> and password file <none>
I20260923 20:14:47.172828 24034 fs_manager.cc:359] Metadata directory not provided
I20260923 20:14:47.172987 24034 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:14:47.173384 24034 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:14:47.178189 24034 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790194121577027-1706-0/minicluster-data/master-0/data/instance:
uuid: "29c11af02b1e4ad6986aa36a52c8229d"
format_stamp: "Formatted at 2026-09-23 20:14:47 on dist-test-slave-th4g"
server_key: "111321d887938cb75f47c39e3209bf23"
server_key_iv: "fdb906d78bfc9da0de22a03f7b1e97e8"
server_key_version: "encryptionkey@0"
I20260923 20:14:47.179442 24034 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790194121577027-1706-0/minicluster-data/master-0/wal/instance:
uuid: "29c11af02b1e4ad6986aa36a52c8229d"
format_stamp: "Formatted at 2026-09-23 20:14:47 on dist-test-slave-th4g"
server_key: "111321d887938cb75f47c39e3209bf23"
server_key_iv: "fdb906d78bfc9da0de22a03f7b1e97e8"
server_key_version: "encryptionkey@0"
I20260923 20:14:47.186905 24034 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.002s sys 0.005s
I20260923 20:14:47.190814 24049 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:14:47.192199 24034 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260923 20:14:47.192401 24034 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790194121577027-1706-0/minicluster-data/master-0/data,/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790194121577027-1706-0/minicluster-data/master-0/wal
uuid: "29c11af02b1e4ad6986aa36a52c8229d"
format_stamp: "Formatted at 2026-09-23 20:14:47 on dist-test-slave-th4g"
server_key: "111321d887938cb75f47c39e3209bf23"
server_key_iv: "fdb906d78bfc9da0de22a03f7b1e97e8"
server_key_version: "encryptionkey@0"
I20260923 20:14:47.192612 24034 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790194121577027-1706-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790194121577027-1706-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790194121577027-1706-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:14:47.198624 24034 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:14:47.203629 24034 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:14:47.203960 24034 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:14:47.223635 24034 rpc_server.cc:307] RPC server started. Bound to: 127.1.170.190:33941
I20260923 20:14:47.223711 24100 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.170.190:33941 every 8 connection(s)
I20260923 20:14:47.225019 24034 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790194121577027-1706-0/minicluster-data/master-0/data/info.pb
I20260923 20:14:47.228820 24101 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:14:47.233620 1706 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqUuku6/build/asan/bin/kudu as pid 24034
I20260923 20:14:47.233814 1706 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790194121577027-1706-0/minicluster-data/master-0/wal/instance
I20260923 20:14:47.234369 1706 external_mini_cluster.cc:1428] Setting key 3b390bf2adb9a69d756de9b418239509
I20260923 20:14:47.240861 24101 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 29c11af02b1e4ad6986aa36a52c8229d: Bootstrap starting.
I20260923 20:14:47.243185 24104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:14:47.236340 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51437 (local address 127.1.170.190:33941)
0923 20:14:47.237318 (+ 978us) server_negotiation.cc:206] Beginning negotiation
0923 20:14:47.237332 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0923 20:14:47.237376 (+ 44us) server_negotiation.cc:407] Connection header received
0923 20:14:47.239516 (+ 2140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:14:47.239541 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:14:47.239919 (+ 378us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:14:47.240267 (+ 348us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:14:47.241412 (+ 1145us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:14:47.241423 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:14:47.241437 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 20:14:47.241456 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:14:47.241519 (+ 63us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 20:14:47.241541 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:14:47.241770 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0923 20:14:47.241943 (+ 173us) server_negotiation.cc:299] Negotiation successful
0923 20:14:47.242168 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":733,"thread_start_us":470,"threads_started":1}
I20260923 20:14:47.244709 24101 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 29c11af02b1e4ad6986aa36a52c8229d: Neither blocks nor log segments found. Creating new log.
I20260923 20:14:47.246028 24101 log.cc:824] T 00000000000000000000000000000000 P 29c11af02b1e4ad6986aa36a52c8229d: Log is configured to *not* fsync() on all Append() calls
I20260923 20:14:47.248876 1706 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 20:14:47.250154 24101 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 29c11af02b1e4ad6986aa36a52c8229d: No bootstrap required, opened a new log
I20260923 20:14:47.257493 24104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:14:47.254785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37999 (local address 127.1.170.190:33941)
0923 20:14:47.255023 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0923 20:14:47.255031 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 20:14:47.255506 (+ 475us) server_negotiation.cc:407] Connection header received
0923 20:14:47.255655 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:14:47.255661 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:14:47.255726 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:14:47.255836 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:14:47.256822 (+ 986us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:14:47.256830 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:14:47.256835 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 20:14:47.256855 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:14:47.256904 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 20:14:47.256929 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:14:47.257066 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0923 20:14:47.257117 (+ 51us) server_negotiation.cc:299] Negotiation successful
0923 20:14:47.257181 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260923 20:14:47.260829 24101 raft_consensus.cc:348] T 00000000000000000000000000000000 P 29c11af02b1e4ad6986aa36a52c8229d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29c11af02b1e4ad6986aa36a52c8229d" member_type: VOTER last_known_addr { host: "127.1.170.190" port: 33941 } }
I20260923 20:14:47.261211 24101 raft_consensus.cc:374] T 00000000000000000000000000000000 P 29c11af02b1e4ad6986aa36a52c8229d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:14:47.261277 24101 raft_consensus.cc:729] T 00000000000000000000000000000000 P 29c11af02b1e4ad6986aa36a52c8229d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29c11af02b1e4ad6986aa36a52c8229d, State: Initialized, Role: FOLLOWER
I20260923 20:14:47.261780 24101 consensus_queue.cc:260] T 00000000000000000000000000000000 P 29c11af02b1e4ad6986aa36a52c8229d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29c11af02b1e4ad6986aa36a52c8229d" member_type: VOTER last_known_addr { host: "127.1.170.190" port: 33941 } }
I20260923 20:14:47.262090 24101 raft_consensus.cc:388] T 00000000000000000000000000000000 P 29c11af02b1e4ad6986aa36a52c8229d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 20:14:47.262266 24101 raft_consensus.cc:482] T 00000000000000000000000000000000 P 29c11af02b1e4ad6986aa36a52c8229d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 20:14:47.262468 24101 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 29c11af02b1e4ad6986aa36a52c8229d [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:14:47.264829 24101 raft_consensus.cc:504] T 00000000000000000000000000000000 P 29c11af02b1e4ad6986aa36a52c8229d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29c11af02b1e4ad6986aa36a52c8229d" member_type: VOTER last_known_addr { host: "127.1.170.190" port: 33941 } }
I20260923 20:14:47.265393 24101 leader_election.cc:302] T 00000000000000000000000000000000 P 29c11af02b1e4ad6986aa36a52c8229d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 29c11af02b1e4ad6986aa36a52c8229d; no voters:
I20260923 20:14:47.266070 24101 leader_election.cc:288] T 00000000000000000000000000000000 P 29c11af02b1e4ad6986aa36a52c8229d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 20:14:47.266188 24111 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 29c11af02b1e4ad6986aa36a52c8229d [term 1 FOLLOWER]: Leader election won for term 1
I20260923 20:14:47.267527 24111 raft_consensus.cc:686] T 00000000000000000000000000000000 P 29c11af02b1e4ad6986aa36a52c8229d [term 1 LEADER]: Becoming Leader. State: Replica: 29c11af02b1e4ad6986aa36a52c8229d, State: Running, Role: LEADER
I20260923 20:14:47.268066 24111 consensus_queue.cc:237] T 00000000000000000000000000000000 P 29c11af02b1e4ad6986aa36a52c8229d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29c11af02b1e4ad6986aa36a52c8229d" member_type: VOTER last_known_addr { host: "127.1.170.190" port: 33941 } }
I20260923 20:14:47.268438 24101 sys_catalog.cc:564] T 00000000000000000000000000000000 P 29c11af02b1e4ad6986aa36a52c8229d [sys.catalog]: configured and running, proceeding with master startup.
I20260923 20:14:47.271121 24113 sys_catalog.cc:455] T 00000000000000000000000000000000 P 29c11af02b1e4ad6986aa36a52c8229d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 29c11af02b1e4ad6986aa36a52c8229d. Latest consensus state: current_term: 1 leader_uuid: "29c11af02b1e4ad6986aa36a52c8229d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29c11af02b1e4ad6986aa36a52c8229d" member_type: VOTER last_known_addr { host: "127.1.170.190" port: 33941 } } }
I20260923 20:14:47.271677 24113 sys_catalog.cc:458] T 00000000000000000000000000000000 P 29c11af02b1e4ad6986aa36a52c8229d [sys.catalog]: This master's current role is: LEADER
I20260923 20:14:47.271250 24112 sys_catalog.cc:455] T 00000000000000000000000000000000 P 29c11af02b1e4ad6986aa36a52c8229d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "29c11af02b1e4ad6986aa36a52c8229d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29c11af02b1e4ad6986aa36a52c8229d" member_type: VOTER last_known_addr { host: "127.1.170.190" port: 33941 } } }
I20260923 20:14:47.271984 24112 sys_catalog.cc:458] T 00000000000000000000000000000000 P 29c11af02b1e4ad6986aa36a52c8229d [sys.catalog]: This master's current role is: LEADER
I20260923 20:14:47.273512 24120 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 20:14:47.278214 24120 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 20:14:47.288440 24120 catalog_manager.cc:1347] Generated new cluster ID: d9b89eab604149afaeb7fcb5e05a44c2
I20260923 20:14:47.288575 24120 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 20:14:47.309506 24120 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 20:14:47.310070 24120 catalog_manager.cc:1504] Loading token signing keys...
I20260923 20:14:47.325522 24120 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 29c11af02b1e4ad6986aa36a52c8229d: Generated new TSK 0
I20260923 20:14:47.326685 24120 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 20:14:47.341180 1706 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskqUuku6/build/asan/bin/kudu pid 24034
W20260923 20:14:47.343677 24096 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:51437
W20260923 20:14:47.344467 1706 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskqUuku6/build/asan/bin/kudu pid 24034: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260923 20:14:47.344650 1706 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqUuku6/build/asan/bin/kudu with pid 24034
2026-09-23T20:14:47Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1981 ms)
[----------] 18 tests from AuthTokenIssuingTest (36914 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790194121577027-1706-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 20:14:47 dist-test-slave-th4g krb5kdc[24132](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 20:14:47 dist-test-slave-th4g krb5kdc[24132](info): set up 2 sockets
Sep 23 20:14:47 dist-test-slave-th4g krb5kdc[24132](info): commencing operation
krb5kdc: starting...
W20260923 20:14:48.478875 1706 mini_kdc.cc:121] Time spent starting KDC: real 1.097s user 0.002s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790194121577027-1706-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790194121577027-1706-0/krb5kdc/test-user.keytab.
Sep 23 20:14:48 dist-test-slave-th4g krb5kdc[24132](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194488, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T20:14:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T20:14:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.170.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.170.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.170.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790194121577027-1706-0/krb5kdc/kudu_127.1.170.190.keytab.
Entry for principal kudu/127.1.170.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790194121577027-1706-0/krb5kdc/kudu_127.1.170.190.keytab.
I20260923 20:14:48.761790 1706 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqUuku6/build/asan/bin/kudu
/tmp/dist-test-taskqUuku6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790194121577027-1706-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790194121577027-1706-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790194121577027-1706-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790194121577027-1706-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.170.190:39017
--webserver_interface=127.1.170.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790194121577027-1706-0/krb5kdc/kudu_127.1.170.190.keytab
--principal=kudu/127.1.170.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.170.148:45243
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.170.190:39017,127.1.170.189:32957,127.1.170.188:43729
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790194121577027-1706-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790194121577027-1706-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790194121577027-1706-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:14:49.094365 24145 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260923 20:14:49.094735 24145 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:14:49.094870 24145 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:14:49.094991 24145 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:14:49.102363 24145 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 20:14:49.102454 24145 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:14:49.102564 24145 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:14:49.102653 24145 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 20:14:49.102746 24145 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 20:14:49.112641 24145 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.170.148:45243
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790194121577027-1706-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790194121577027-1706-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.170.190:39017,127.1.170.189:32957,127.1.170.188:43729
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790194121577027-1706-0/krb5kdc/kudu_127.1.170.190.keytab
--principal=kudu/127.1.170.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.170.190:39017
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790194121577027-1706-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.170.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790194121577027-1706-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:14:49.114008 24145 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:14:49.115311 24145 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:14:49.127125 24150 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:14:49.127081 24151 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:14:49.127187 24153 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:14:49.128471 24145 server_base.cc:1034] running on GCE node
I20260923 20:14:49.129249 24145 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:14:49.130743 24145 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:14:49.131992 24145 hybrid_clock.cc:648] HybridClock initialized: now 1790194489131939 us; error 32 us; skew 500 ppm
I20260923 20:14:49.132329 24145 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:14:49 dist-test-slave-th4g krb5kdc[24132](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194489, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.170.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:14:49.142489 24145 init.cc:377] Logged in from keytab as kudu/127.1.170.190@KRBTEST.COM (short username kudu)
I20260923 20:14:49.143972 24145 webserver.cc:460] Webserver started at http://127.1.170.190:45943/ using document root <none> and password file <none>
I20260923 20:14:49.144637 24145 fs_manager.cc:359] Metadata directory not provided
I20260923 20:14:49.144745 24145 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:14:49.145171 24145 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:14:49.148250 24145 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790194121577027-1706-0/minicluster-data/master-0/data/instance:
uuid: "405a4d054e444ce99c6a927df9098557"
format_stamp: "Formatted at 2026-09-23 20:14:49 on dist-test-slave-th4g"
server_key: "3271b0497c044a45717a2394dc5944e3"
server_key_iv: "11a77222c42c80d23990b0234ba98b07"
server_key_version: "encryptionkey@0"
I20260923 20:14:49.149034 24145 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790194121577027-1706-0/minicluster-data/master-0/wal/instance:
uuid: "405a4d054e444ce99c6a927df9098557"
format_stamp: "Formatted at 2026-09-23 20:14:49 on dist-test-slave-th4g"
server_key: "3271b0497c044a45717a2394dc5944e3"
server_key_iv: "11a77222c42c80d23990b0234ba98b07"
server_key_version: "encryptionkey@0"
I20260923 20:14:49.154363 24145 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260923 20:14:49.158015 24160 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:14:49.159340 24145 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260923 20:14:49.159535 24145 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790194121577027-1706-0/minicluster-data/master-0/data,/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790194121577027-1706-0/minicluster-data/master-0/wal
uuid: "405a4d054e444ce99c6a927df9098557"
format_stamp: "Formatted at 2026-09-23 20:14:49 on dist-test-slave-th4g"
server_key: "3271b0497c044a45717a2394dc5944e3"
server_key_iv: "11a77222c42c80d23990b0234ba98b07"
server_key_version: "encryptionkey@0"
I20260923 20:14:49.159765 24145 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790194121577027-1706-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790194121577027-1706-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790194121577027-1706-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:14:49.183845 24145 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:14:49.188181 24145 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:14:49.188473 24145 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:14:49.207643 24145 rpc_server.cc:307] RPC server started. Bound to: 127.1.170.190:39017
I20260923 20:14:49.207728 24211 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.170.190:39017 every 8 connection(s)
I20260923 20:14:49.209002 24145 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790194121577027-1706-0/minicluster-data/master-0/data/info.pb
I20260923 20:14:49.212667 24212 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:14:49.218369 1706 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqUuku6/build/asan/bin/kudu as pid 24145
I20260923 20:14:49.218677 1706 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790194121577027-1706-0/minicluster-data/master-0/wal/instance
I20260923 20:14:49.219324 1706 external_mini_cluster.cc:1428] Setting key 185b9a63562e606f5b5009bef6736ec9
I20260923 20:14:49.219153 24212 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.170.190" port: 39017 } has no permanent_uuid. Determining permanent_uuid...
Sep 23 20:14:49 dist-test-slave-th4g krb5kdc[24132](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194489, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.170.190@KRBTEST.COM for kudu/127.1.170.190@KRBTEST.COM
I20260923 20:14:49.242129 24214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:14:49.222267 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.190:39017 (local address 127.0.0.1:33544)
0923 20:14:49.223099 (+ 832us) negotiation.cc:107] Waiting for socket to connect
0923 20:14:49.223141 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0923 20:14:49.225041 (+ 1900us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:14:49.225836 (+ 795us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:14:49.225842 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:14:49.226580 (+ 738us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:14:49.226898 (+ 318us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:14:49.226915 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:14:49.228430 (+ 1515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:14:49.228439 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:14:49.229460 (+ 1021us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:14:49.229472 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:14:49.230435 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:14:49.230441 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:14:49.230612 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:14:49.231395 (+ 783us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:14:49.231418 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:14:49.233822 (+ 2404us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:14:49.238155 (+ 4333us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:14:49.238165 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:14:49.238175 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:14:49.238583 (+ 408us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:14:49.239071 (+ 488us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:14:49.239074 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:14:49.239076 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:14:49.239227 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:14:49.240235 (+ 1008us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:14:49.240241 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:14:49.240424 (+ 183us) client_negotiation.cc:769] Sending connection context
0923 20:14:49.240633 (+ 209us) client_negotiation.cc:240] Negotiation successful
0923 20:14:49.241042 (+ 409us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":574,"thread_start_us":365,"threads_started":1}
I20260923 20:14:49.242179 24215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:14:49.222267 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33544 (local address 127.1.170.190:39017)
0923 20:14:49.223846 (+ 1579us) server_negotiation.cc:206] Beginning negotiation
0923 20:14:49.223854 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 20:14:49.223906 (+ 52us) server_negotiation.cc:407] Connection header received
0923 20:14:49.225400 (+ 1494us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:14:49.225405 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:14:49.225624 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:14:49.225822 (+ 198us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:14:49.227109 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:14:49.228272 (+ 1163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:14:49.229613 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:14:49.230206 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:14:49.230395 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:14:49.234132 (+ 3737us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:14:49.234145 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:14:49.234163 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:14:49.234205 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:14:49.238000 (+ 3795us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:14:49.238739 (+ 739us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:14:49.238750 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:14:49.238757 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:14:49.238956 (+ 199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:14:49.239352 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:14:49.239360 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:14:49.239366 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:14:49.240093 (+ 727us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:14:49.240310 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0923 20:14:49.240864 (+ 554us) server_negotiation.cc:299] Negotiation successful
0923 20:14:49.241262 (+ 398us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":58,"server-negotiator.queue_time_us":1366,"thread_start_us":1143,"threads_started":1}
WARNING: no policy specified for kudu/127.1.170.189@KRBTEST.COM; defaulting to no policy
I20260923 20:14:49.245398 24212 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.170.189" port: 32957 } has no permanent_uuid. Determining permanent_uuid...
I20260923 20:14:49.246562 24214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:14:49.246108 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.189:32957 (local address 127.0.0.1:39292)
0923 20:14:49.246328 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0923 20:14:49.246439 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.170.189:32957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
W20260923 20:14:49.246797 24164 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.170.189:32957: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.170.189@KRBTEST.COM" created.
I20260923 20:14:49.248839 24214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:14:49.248340 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.189:32957 (local address 127.0.0.1:39306)
0923 20:14:49.248545 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0923 20:14:49.248655 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.170.189:32957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
W20260923 20:14:49.249082 24212 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.170.189:32957: Network error: Client connection negotiation failed: client connection to 127.1.170.189:32957: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.170.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790194121577027-1706-0/krb5kdc/kudu_127.1.170.189.keytab.
Entry for principal kudu/127.1.170.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790194121577027-1706-0/krb5kdc/kudu_127.1.170.189.keytab.
I20260923 20:14:49.278504 1706 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqUuku6/build/asan/bin/kudu
/tmp/dist-test-taskqUuku6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790194121577027-1706-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790194121577027-1706-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790194121577027-1706-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790194121577027-1706-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.170.189:32957
--webserver_interface=127.1.170.189
--webserver_port=0
--keytab_file=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790194121577027-1706-0/krb5kdc/kudu_127.1.170.189.keytab
--principal=kudu/127.1.170.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.170.148:45243
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.170.190:39017,127.1.170.189:32957,127.1.170.188:43729
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790194121577027-1706-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790194121577027-1706-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790194121577027-1706-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260923 20:14:49.298461 24212 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.170.189" port: 32957 } attempt: 1
I20260923 20:14:49.299875 24214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:14:49.299254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.189:32957 (local address 127.0.0.1:39316)
0923 20:14:49.299511 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0923 20:14:49.299666 (+ 155us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.170.189:32957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
I20260923 20:14:49.301144 24214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:14:49.300652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.189:32957 (local address 127.0.0.1:39328)
0923 20:14:49.300869 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0923 20:14:49.300966 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.170.189:32957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260923 20:14:49.301383 24212 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.170.189:32957: Network error: Client connection negotiation failed: client connection to 127.1.170.189:32957: connect: Connection refused (error 111)
I20260923 20:14:49.369674 24212 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.170.189" port: 32957 } attempt: 2
I20260923 20:14:49.370954 24214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:14:49.370379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.189:32957 (local address 127.0.0.1:39334)
0923 20:14:49.370672 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0923 20:14:49.370782 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.170.189:32957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103}
I20260923 20:14:49.372058 24214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:14:49.371665 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.189:32957 (local address 127.0.0.1:39348)
0923 20:14:49.371814 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0923 20:14:49.371909 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.170.189:32957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260923 20:14:49.372278 24212 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.170.189:32957: Network error: Client connection negotiation failed: client connection to 127.1.170.189:32957: connect: Connection refused (error 111)
I20260923 20:14:49.463546 24212 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.170.189" port: 32957 } attempt: 3
I20260923 20:14:49.464936 24214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:14:49.464333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.189:32957 (local address 127.0.0.1:39354)
0923 20:14:49.464604 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0923 20:14:49.464735 (+ 131us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.170.189:32957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
I20260923 20:14:49.466290 24214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:14:49.465749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.189:32957 (local address 127.0.0.1:39364)
0923 20:14:49.465973 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0923 20:14:49.466076 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.170.189:32957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
W20260923 20:14:49.466555 24212 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.170.189:32957: Network error: Client connection negotiation failed: client connection to 127.1.170.189:32957: connect: Connection refused (error 111)
W20260923 20:14:49.604339 24219 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260923 20:14:49.604689 24219 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:14:49.604765 24219 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:14:49.604852 24219 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260923 20:14:49.609840 24212 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.170.189" port: 32957 } attempt: 4
I20260923 20:14:49.611083 24214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:14:49.610507 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.189:32957 (local address 127.0.0.1:39374)
0923 20:14:49.610807 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0923 20:14:49.610923 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.170.189:32957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":160}
I20260923 20:14:49.612068 24214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:14:49.611712 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.189:32957 (local address 127.0.0.1:39384)
0923 20:14:49.611851 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0923 20:14:49.611938 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.170.189:32957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260923 20:14:49.612275 24212 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.170.189:32957: Network error: Client connection negotiation failed: client connection to 127.1.170.189:32957: connect: Connection refused (error 111)
W20260923 20:14:49.612884 24219 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 20:14:49.612957 24219 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:14:49.613054 24219 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:14:49.613124 24219 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 20:14:49.613232 24219 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 20:14:49.623190 24219 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.170.148:45243
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790194121577027-1706-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790194121577027-1706-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.1.170.190:39017,127.1.170.189:32957,127.1.170.188:43729
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790194121577027-1706-0/krb5kdc/kudu_127.1.170.189.keytab
--principal=kudu/127.1.170.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.170.189:32957
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790194121577027-1706-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.1.170.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790194121577027-1706-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:14:49.624464 24219 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:14:49.625697 24219 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:14:49.637086 24224 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:14:49.637059 24225 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:14:49.637146 24227 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:14:49.638198 24219 server_base.cc:1034] running on GCE node
I20260923 20:14:49.638906 24219 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:14:49.640192 24219 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:14:49.641412 24219 hybrid_clock.cc:648] HybridClock initialized: now 1790194489641356 us; error 31 us; skew 500 ppm
I20260923 20:14:49.641746 24219 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:14:49 dist-test-slave-th4g krb5kdc[24132](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194489, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.170.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:14:49.644829 24219 init.cc:377] Logged in from keytab as kudu/127.1.170.189@KRBTEST.COM (short username kudu)
I20260923 20:14:49.646252 24219 webserver.cc:460] Webserver started at http://127.1.170.189:43931/ using document root <none> and password file <none>
I20260923 20:14:49.646901 24219 fs_manager.cc:359] Metadata directory not provided
I20260923 20:14:49.647027 24219 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:14:49.647343 24219 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:14:49.650241 24219 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790194121577027-1706-0/minicluster-data/master-1/data/instance:
uuid: "657c5e3e229e4b6b95203d69fae92453"
format_stamp: "Formatted at 2026-09-23 20:14:49 on dist-test-slave-th4g"
server_key: "f2c271efd5ffbc1267f1377de07a8adc"
server_key_iv: "bfbec9f489bc3b956e3b17fe2a1cd308"
server_key_version: "encryptionkey@0"
I20260923 20:14:49.651005 24219 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790194121577027-1706-0/minicluster-data/master-1/wal/instance:
uuid: "657c5e3e229e4b6b95203d69fae92453"
format_stamp: "Formatted at 2026-09-23 20:14:49 on dist-test-slave-th4g"
server_key: "f2c271efd5ffbc1267f1377de07a8adc"
server_key_iv: "bfbec9f489bc3b956e3b17fe2a1cd308"
server_key_version: "encryptionkey@0"
I20260923 20:14:49.656081 24219 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260923 20:14:49.659446 24234 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:14:49.660773 24219 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260923 20:14:49.660952 24219 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790194121577027-1706-0/minicluster-data/master-1/data,/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790194121577027-1706-0/minicluster-data/master-1/wal
uuid: "657c5e3e229e4b6b95203d69fae92453"
format_stamp: "Formatted at 2026-09-23 20:14:49 on dist-test-slave-th4g"
server_key: "f2c271efd5ffbc1267f1377de07a8adc"
server_key_iv: "bfbec9f489bc3b956e3b17fe2a1cd308"
server_key_version: "encryptionkey@0"
I20260923 20:14:49.661175 24219 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790194121577027-1706-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790194121577027-1706-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790194121577027-1706-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:14:49.675513 24219 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:14:49.679569 24219 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:14:49.679847 24219 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:14:49.698544 24219 rpc_server.cc:307] RPC server started. Bound to: 127.1.170.189:32957
I20260923 20:14:49.698583 24285 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.170.189:32957 every 8 connection(s)
I20260923 20:14:49.699898 24219 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790194121577027-1706-0/minicluster-data/master-1/data/info.pb
I20260923 20:14:49.702497 1706 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqUuku6/build/asan/bin/kudu as pid 24219
I20260923 20:14:49.702710 1706 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790194121577027-1706-0/minicluster-data/master-1/wal/instance
I20260923 20:14:49.703145 1706 external_mini_cluster.cc:1428] Setting key d8e85bc5ffd596384ddb1d57ca50a0f6
I20260923 20:14:49.703588 24286 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:14:49.709830 24286 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.170.190" port: 39017 } has no permanent_uuid. Determining permanent_uuid...
Sep 23 20:14:49 dist-test-slave-th4g krb5kdc[24132](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194489, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.170.189@KRBTEST.COM for kudu/127.1.170.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.1.170.188@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.170.188@KRBTEST.COM" created.
I20260923 20:14:49.731501 24215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:14:49.712732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33548 (local address 127.1.170.190:39017)
0923 20:14:49.712972 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0923 20:14:49.712981 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 20:14:49.714061 (+ 1080us) server_negotiation.cc:407] Connection header received
0923 20:14:49.715936 (+ 1875us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:14:49.715945 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:14:49.716040 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:14:49.716156 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:14:49.717673 (+ 1517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:14:49.718848 (+ 1175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:14:49.720419 (+ 1571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:14:49.721220 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:14:49.721364 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:14:49.725137 (+ 3773us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:14:49.725146 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:14:49.725154 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:14:49.725199 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:14:49.728270 (+ 3071us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:14:49.729088 (+ 818us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:14:49.729093 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:14:49.729096 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:14:49.729260 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:14:49.729683 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:14:49.729690 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:14:49.729693 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:14:49.730052 (+ 359us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:14:49.730204 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0923 20:14:49.730941 (+ 737us) server_negotiation.cc:299] Negotiation successful
0923 20:14:49.731167 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260923 20:14:49.732463 24289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:14:49.713091 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.190:39017 (local address 127.0.0.1:33548)
0923 20:14:49.713895 (+ 804us) negotiation.cc:107] Waiting for socket to connect
0923 20:14:49.713939 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0923 20:14:49.715669 (+ 1730us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:14:49.716385 (+ 716us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:14:49.716393 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:14:49.716950 (+ 557us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:14:49.717489 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:14:49.717505 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:14:49.719027 (+ 1522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:14:49.719031 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:14:49.720254 (+ 1223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:14:49.720266 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:14:49.721423 (+ 1157us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:14:49.721428 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:14:49.721627 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:14:49.722542 (+ 915us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:14:49.722587 (+ 45us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:14:49.724898 (+ 2311us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:14:49.728437 (+ 3539us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:14:49.728446 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:14:49.728464 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:14:49.728956 (+ 492us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:14:49.729377 (+ 421us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:14:49.729382 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:14:49.729386 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:14:49.729568 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:14:49.730180 (+ 612us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:14:49.730188 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:14:49.730568 (+ 380us) client_negotiation.cc:769] Sending connection context
0923 20:14:49.730848 (+ 280us) client_negotiation.cc:240] Negotiation successful
0923 20:14:49.731354 (+ 506us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":534,"thread_start_us":341,"threads_started":1}
I20260923 20:14:49.734302 24286 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.170.189" port: 32957 } has no permanent_uuid. Determining permanent_uuid...
Sep 23 20:14:49 dist-test-slave-th4g krb5kdc[24132](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194489, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.170.189@KRBTEST.COM for kudu/127.1.170.189@KRBTEST.COM
I20260923 20:14:49.752661 24289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:14:49.735142 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.189:32957 (local address 127.0.0.1:39396)
0923 20:14:49.735390 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0923 20:14:49.735413 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0923 20:14:49.735521 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:14:49.736506 (+ 985us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:14:49.736517 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:14:49.737037 (+ 520us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:14:49.737274 (+ 237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:14:49.737290 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:14:49.739325 (+ 2035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:14:49.739333 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:14:49.740963 (+ 1630us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:14:49.740980 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:14:49.741843 (+ 863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:14:49.741852 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:14:49.742149 (+ 297us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:14:49.742800 (+ 651us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:14:49.742827 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:14:49.745401 (+ 2574us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:14:49.749983 (+ 4582us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:14:49.749987 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:14:49.749990 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:14:49.750382 (+ 392us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:14:49.750837 (+ 455us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:14:49.750840 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:14:49.750843 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:14:49.750982 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:14:49.751757 (+ 775us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:14:49.751760 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:14:49.751886 (+ 126us) client_negotiation.cc:769] Sending connection context
0923 20:14:49.752075 (+ 189us) client_negotiation.cc:240] Negotiation successful
0923 20:14:49.752335 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":73}
I20260923 20:14:49.753113 24290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:14:49.735350 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39396 (local address 127.1.170.189:32957)
0923 20:14:49.736098 (+ 748us) server_negotiation.cc:206] Beginning negotiation
0923 20:14:49.736104 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:14:49.736127 (+ 23us) server_negotiation.cc:407] Connection header received
0923 20:14:49.736168 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:14:49.736173 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:14:49.736341 (+ 168us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:14:49.736469 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:14:49.737462 (+ 993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:14:49.739179 (+ 1717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:14:49.741150 (+ 1971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:14:49.741684 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:14:49.741785 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:14:49.745627 (+ 3842us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:14:49.745644 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:14:49.745663 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:14:49.745705 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:14:49.749746 (+ 4041us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:14:49.750523 (+ 777us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:14:49.750535 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:14:49.750542 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:14:49.750697 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:14:49.751093 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:14:49.751100 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:14:49.751106 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:14:49.751594 (+ 488us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:14:49.751799 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0923 20:14:49.752272 (+ 473us) server_negotiation.cc:299] Negotiation successful
0923 20:14:49.752551 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":571,"thread_start_us":422,"threads_started":1}
I20260923 20:14:49.756053 24286 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.170.188" port: 43729 } has no permanent_uuid. Determining permanent_uuid...
I20260923 20:14:49.757468 24289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:14:49.756875 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.188:43729 (local address 127.0.0.1:48648)
0923 20:14:49.757115 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0923 20:14:49.757263 (+ 148us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.170.188:43729: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
W20260923 20:14:49.757732 24237 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.170.188:43729: connect: Connection refused (error 111)
I20260923 20:14:49.760097 24289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:14:49.759738 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.188:43729 (local address 127.0.0.1:48660)
0923 20:14:49.759915 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0923 20:14:49.759994 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.170.188:43729: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260923 20:14:49.760470 24286 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.170.188:43729: Network error: Client connection negotiation failed: client connection to 127.1.170.188:43729: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.170.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790194121577027-1706-0/krb5kdc/kudu_127.1.170.188.keytab.
Entry for principal kudu/127.1.170.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790194121577027-1706-0/krb5kdc/kudu_127.1.170.188.keytab.
I20260923 20:14:49.769232 1706 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqUuku6/build/asan/bin/kudu
/tmp/dist-test-taskqUuku6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790194121577027-1706-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790194121577027-1706-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790194121577027-1706-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790194121577027-1706-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.170.188:43729
--webserver_interface=127.1.170.188
--webserver_port=0
--keytab_file=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790194121577027-1706-0/krb5kdc/kudu_127.1.170.188.keytab
--principal=kudu/127.1.170.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.170.148:45243
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.170.190:39017,127.1.170.189:32957,127.1.170.188:43729
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790194121577027-1706-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790194121577027-1706-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790194121577027-1706-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260923 20:14:49.809913 24286 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.170.188" port: 43729 } attempt: 1
I20260923 20:14:49.811164 24289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:14:49.810671 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.188:43729 (local address 127.0.0.1:48676)
0923 20:14:49.810893 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0923 20:14:49.810994 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.170.188:43729: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96}
I20260923 20:14:49.812147 24289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:14:49.811808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.188:43729 (local address 127.0.0.1:48690)
0923 20:14:49.811969 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0923 20:14:49.812039 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.170.188:43729: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260923 20:14:49.812350 24286 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.170.188:43729: Network error: Client connection negotiation failed: client connection to 127.1.170.188:43729: connect: Connection refused (error 111)
I20260923 20:14:49.880610 24286 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.170.188" port: 43729 } attempt: 2
I20260923 20:14:49.881775 24289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:14:49.881271 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.188:43729 (local address 127.0.0.1:48700)
0923 20:14:49.881465 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0923 20:14:49.881561 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.170.188:43729: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260923 20:14:49.882750 24289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:14:49.882420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.188:43729 (local address 127.0.0.1:48706)
0923 20:14:49.882566 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0923 20:14:49.882645 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.170.188:43729: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260923 20:14:49.882936 24286 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.170.188:43729: Network error: Client connection negotiation failed: client connection to 127.1.170.188:43729: connect: Connection refused (error 111)
I20260923 20:14:49.911572 24212 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.170.189" port: 32957 } attempt: 5
Sep 23 20:14:49 dist-test-slave-th4g krb5kdc[24132](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194489, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.170.190@KRBTEST.COM for kudu/127.1.170.189@KRBTEST.COM
I20260923 20:14:49.929140 24290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:14:49.912484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39402 (local address 127.1.170.189:32957)
0923 20:14:49.912694 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0923 20:14:49.912700 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:14:49.912725 (+ 25us) server_negotiation.cc:407] Connection header received
0923 20:14:49.912889 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:14:49.912897 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:14:49.913016 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:14:49.913160 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:14:49.913977 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:14:49.915148 (+ 1171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:14:49.916453 (+ 1305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:14:49.917148 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:14:49.917276 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:14:49.921679 (+ 4403us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:14:49.921686 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:14:49.921691 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:14:49.921737 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:14:49.925799 (+ 4062us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:14:49.926902 (+ 1103us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:14:49.926908 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:14:49.926912 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:14:49.927055 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:14:49.927515 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:14:49.927521 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:14:49.927525 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:14:49.927843 (+ 318us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:14:49.928021 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0923 20:14:49.928517 (+ 496us) server_negotiation.cc:299] Negotiation successful
0923 20:14:49.928764 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260923 20:14:49.929162 24214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:14:49.912371 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.189:32957 (local address 127.0.0.1:39402)
0923 20:14:49.912592 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0923 20:14:49.912618 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0923 20:14:49.912750 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:14:49.913204 (+ 454us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:14:49.913208 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:14:49.913588 (+ 380us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:14:49.913792 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:14:49.913801 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:14:49.915299 (+ 1498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:14:49.915304 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:14:49.916269 (+ 965us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:14:49.916282 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:14:49.917298 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:14:49.917303 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:14:49.917479 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:14:49.918228 (+ 749us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:14:49.918247 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:14:49.921420 (+ 3173us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:14:49.926055 (+ 4635us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:14:49.926062 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:14:49.926066 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:14:49.926774 (+ 708us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:14:49.927186 (+ 412us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:14:49.927192 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:14:49.927197 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:14:49.927396 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:14:49.927994 (+ 598us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:14:49.928000 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:14:49.928198 (+ 198us) client_negotiation.cc:769] Sending connection context
0923 20:14:49.928409 (+ 211us) client_negotiation.cc:240] Negotiation successful
0923 20:14:49.928754 (+ 345us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":63}
I20260923 20:14:49.930595 24212 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.170.188" port: 43729 } has no permanent_uuid. Determining permanent_uuid...
I20260923 20:14:49.931948 24214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:14:49.931447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.188:43729 (local address 127.0.0.1:48716)
0923 20:14:49.931647 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0923 20:14:49.931766 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.170.188:43729: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260923 20:14:49.933015 24214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:14:49.932639 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.188:43729 (local address 127.0.0.1:48732)
0923 20:14:49.932805 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0923 20:14:49.932896 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.170.188:43729: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260923 20:14:49.933259 24212 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.170.188:43729: Network error: Client connection negotiation failed: client connection to 127.1.170.188:43729: connect: Connection refused (error 111)
I20260923 20:14:49.974159 24286 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.170.188" port: 43729 } attempt: 3
I20260923 20:14:49.975359 24289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:14:49.974829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.188:43729 (local address 127.0.0.1:48740)
0923 20:14:49.975062 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0923 20:14:49.975159 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.170.188:43729: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260923 20:14:49.977927 24289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:14:49.977356 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.188:43729 (local address 127.0.0.1:48750)
0923 20:14:49.977624 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0923 20:14:49.977737 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.170.188:43729: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":147}
W20260923 20:14:49.978205 24286 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.170.188:43729: Network error: Client connection negotiation failed: client connection to 127.1.170.188:43729: connect: Connection refused (error 111)
I20260923 20:14:49.984498 24212 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.170.188" port: 43729 } attempt: 1
I20260923 20:14:49.985854 24214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:14:49.985341 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.188:43729 (local address 127.0.0.1:48762)
0923 20:14:49.985553 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0923 20:14:49.985658 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.170.188:43729: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260923 20:14:49.987653 24214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:14:49.987278 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.188:43729 (local address 127.0.0.1:48766)
0923 20:14:49.987440 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0923 20:14:49.987523 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.170.188:43729: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260923 20:14:49.987879 24212 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.170.188:43729: Network error: Client connection negotiation failed: client connection to 127.1.170.188:43729: connect: Connection refused (error 111)
I20260923 20:14:50.056149 24212 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.170.188" port: 43729 } attempt: 2
I20260923 20:14:50.057457 24214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:14:50.056842 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.188:43729 (local address 127.0.0.1:48780)
0923 20:14:50.057100 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0923 20:14:50.057226 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.170.188:43729: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
I20260923 20:14:50.058524 24214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:14:50.058138 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.188:43729 (local address 127.0.0.1:48782)
0923 20:14:50.058305 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0923 20:14:50.058387 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.170.188:43729: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260923 20:14:50.058735 24212 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.170.188:43729: Network error: Client connection negotiation failed: client connection to 127.1.170.188:43729: connect: Connection refused (error 111)
W20260923 20:14:50.112290 24293 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260923 20:14:50.112666 24293 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:14:50.112790 24293 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:14:50.112972 24293 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:14:50.120294 24293 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 20:14:50.120385 24293 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:14:50.120483 24293 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:14:50.120561 24293 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 20:14:50.120671 24293 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 20:14:50.121438 24286 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.170.188" port: 43729 } attempt: 4
I20260923 20:14:50.122751 24289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:14:50.122179 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.188:43729 (local address 127.0.0.1:48784)
0923 20:14:50.122445 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0923 20:14:50.122552 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.170.188:43729: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260923 20:14:50.123782 24289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:14:50.123402 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.188:43729 (local address 127.0.0.1:48786)
0923 20:14:50.123569 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0923 20:14:50.123652 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.170.188:43729: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260923 20:14:50.123988 24286 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.170.188:43729: Network error: Client connection negotiation failed: client connection to 127.1.170.188:43729: connect: Connection refused (error 111)
I20260923 20:14:50.131278 24293 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.170.148:45243
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790194121577027-1706-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790194121577027-1706-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.1.170.190:39017,127.1.170.189:32957,127.1.170.188:43729
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790194121577027-1706-0/krb5kdc/kudu_127.1.170.188.keytab
--principal=kudu/127.1.170.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.170.188:43729
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790194121577027-1706-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.1.170.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790194121577027-1706-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:14:50.132484 24293 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:14:50.133718 24293 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:14:50.145347 24298 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:14:50.145529 24301 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:14:50.145470 24299 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:14:50.146622 24293 server_base.cc:1034] running on GCE node
I20260923 20:14:50.147384 24293 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:14:50.148744 24293 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:14:50.150041 24293 hybrid_clock.cc:648] HybridClock initialized: now 1790194490149970 us; error 35 us; skew 500 ppm
I20260923 20:14:50.150434 24293 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:14:50 dist-test-slave-th4g krb5kdc[24132](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194490, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.170.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:14:50.153555 24293 init.cc:377] Logged in from keytab as kudu/127.1.170.188@KRBTEST.COM (short username kudu)
I20260923 20:14:50.155099 24293 webserver.cc:460] Webserver started at http://127.1.170.188:45353/ using document root <none> and password file <none>
I20260923 20:14:50.155781 24293 fs_manager.cc:359] Metadata directory not provided
I20260923 20:14:50.155893 24293 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:14:50.156211 24293 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:14:50.159323 24293 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790194121577027-1706-0/minicluster-data/master-2/data/instance:
uuid: "04f672b75a424bfea249ca253ea6be46"
format_stamp: "Formatted at 2026-09-23 20:14:50 on dist-test-slave-th4g"
server_key: "221019392b637bc6a417f7cd50277432"
server_key_iv: "81137ded383b5ae00001ca4cce776881"
server_key_version: "encryptionkey@0"
I20260923 20:14:50.160094 24293 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790194121577027-1706-0/minicluster-data/master-2/wal/instance:
uuid: "04f672b75a424bfea249ca253ea6be46"
format_stamp: "Formatted at 2026-09-23 20:14:50 on dist-test-slave-th4g"
server_key: "221019392b637bc6a417f7cd50277432"
server_key_iv: "81137ded383b5ae00001ca4cce776881"
server_key_version: "encryptionkey@0"
I20260923 20:14:50.164940 24212 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.170.188" port: 43729 } attempt: 3
I20260923 20:14:50.165717 24293 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260923 20:14:50.166054 24214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:14:50.165551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.188:43729 (local address 127.0.0.1:48798)
0923 20:14:50.165744 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0923 20:14:50.165856 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.170.188:43729: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260923 20:14:50.167037 24214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:14:50.166707 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.188:43729 (local address 127.0.0.1:48804)
0923 20:14:50.166889 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0923 20:14:50.166949 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.170.188:43729: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92}
W20260923 20:14:50.167333 24212 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.170.188:43729: Network error: Client connection negotiation failed: client connection to 127.1.170.188:43729: connect: Connection refused (error 111)
I20260923 20:14:50.169605 24308 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:14:50.171005 24293 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 20:14:50.171192 24293 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790194121577027-1706-0/minicluster-data/master-2/data,/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790194121577027-1706-0/minicluster-data/master-2/wal
uuid: "04f672b75a424bfea249ca253ea6be46"
format_stamp: "Formatted at 2026-09-23 20:14:50 on dist-test-slave-th4g"
server_key: "221019392b637bc6a417f7cd50277432"
server_key_iv: "81137ded383b5ae00001ca4cce776881"
server_key_version: "encryptionkey@0"
I20260923 20:14:50.171437 24293 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790194121577027-1706-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790194121577027-1706-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790194121577027-1706-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:14:50.197657 24293 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:14:50.202118 24293 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:14:50.202422 24293 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:14:50.221488 24293 rpc_server.cc:307] RPC server started. Bound to: 127.1.170.188:43729
I20260923 20:14:50.221513 24359 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.170.188:43729 every 8 connection(s)
I20260923 20:14:50.222870 24293 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790194121577027-1706-0/minicluster-data/master-2/data/info.pb
I20260923 20:14:50.226420 24360 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:14:50.227517 1706 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqUuku6/build/asan/bin/kudu as pid 24293
I20260923 20:14:50.227737 1706 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790194121577027-1706-0/minicluster-data/master-2/wal/instance
I20260923 20:14:50.228196 1706 external_mini_cluster.cc:1428] Setting key 083a3313014951ec8e3ddde77a0d5e18
I20260923 20:14:50.235707 24360 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.170.190" port: 39017 } has no permanent_uuid. Determining permanent_uuid...
Sep 23 20:14:50 dist-test-slave-th4g krb5kdc[24132](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194488, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.170.188@KRBTEST.COM
Sep 23 20:14:50 dist-test-slave-th4g krb5kdc[24132](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194490, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.170.188@KRBTEST.COM for kudu/127.1.170.190@KRBTEST.COM
I20260923 20:14:50.252478 24363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:14:50.230233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48808 (local address 127.1.170.188:43729)
0923 20:14:50.231027 (+ 794us) server_negotiation.cc:206] Beginning negotiation
0923 20:14:50.231039 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0923 20:14:50.231080 (+ 41us) server_negotiation.cc:407] Connection header received
0923 20:14:50.232610 (+ 1530us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:14:50.232631 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:14:50.232994 (+ 363us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:14:50.233317 (+ 323us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:14:50.235153 (+ 1836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:14:50.236442 (+ 1289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:14:50.238219 (+ 1777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:14:50.239316 (+ 1097us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:14:50.239517 (+ 201us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:14:50.244224 (+ 4707us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:14:50.244231 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:14:50.244241 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:14:50.244273 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:14:50.247706 (+ 3433us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:14:50.248567 (+ 861us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:14:50.248575 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:14:50.248583 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:14:50.248781 (+ 198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:14:50.249150 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:14:50.249158 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:14:50.249161 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:14:50.249909 (+ 748us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:14:50.250231 (+ 322us) server_negotiation.cc:1035] Waiting for connection context
0923 20:14:50.250625 (+ 394us) server_negotiation.cc:299] Negotiation successful
0923 20:14:50.251076 (+ 451us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":590,"thread_start_us":399,"threads_started":1}
I20260923 20:14:50.255242 24365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:14:50.238514 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.190:39017 (local address 127.0.0.1:33560)
0923 20:14:50.239398 (+ 884us) negotiation.cc:107] Waiting for socket to connect
0923 20:14:50.239439 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0923 20:14:50.239570 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:14:50.240046 (+ 476us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:14:50.240053 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:14:50.240703 (+ 650us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:14:50.240935 (+ 232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:14:50.240944 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:14:50.242616 (+ 1672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:14:50.242625 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:14:50.243915 (+ 1290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:14:50.243927 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:14:50.244653 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:14:50.244659 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:14:50.244903 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:14:50.245463 (+ 560us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:14:50.245490 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:14:50.248242 (+ 2752us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:14:50.252294 (+ 4052us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:14:50.252303 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:14:50.252308 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:14:50.252925 (+ 617us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:14:50.253287 (+ 362us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:14:50.253292 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:14:50.253295 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:14:50.253446 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:14:50.254127 (+ 681us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:14:50.254147 (+ 20us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:14:50.254360 (+ 213us) client_negotiation.cc:769] Sending connection context
0923 20:14:50.254562 (+ 202us) client_negotiation.cc:240] Negotiation successful
0923 20:14:50.254847 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":635,"thread_start_us":506,"threads_started":1}
I20260923 20:14:50.255437 24364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:14:50.238472 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33560 (local address 127.1.170.190:39017)
0923 20:14:50.239080 (+ 608us) server_negotiation.cc:206] Beginning negotiation
0923 20:14:50.239086 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:14:50.239671 (+ 585us) server_negotiation.cc:407] Connection header received
0923 20:14:50.239739 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:14:50.239744 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:14:50.239832 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:14:50.239960 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:14:50.241119 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:14:50.242453 (+ 1334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:14:50.244067 (+ 1614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:14:50.244475 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:14:50.244574 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:14:50.248500 (+ 3926us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:14:50.248509 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:14:50.248517 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:14:50.248563 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:14:50.252157 (+ 3594us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:14:50.253055 (+ 898us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:14:50.253059 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:14:50.253061 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:14:50.253179 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:14:50.253593 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:14:50.253597 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:14:50.253600 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:14:50.253979 (+ 379us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:14:50.254138 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0923 20:14:50.254716 (+ 578us) server_negotiation.cc:299] Negotiation successful
0923 20:14:50.254962 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":467,"thread_start_us":304,"threads_started":1}
I20260923 20:14:50.256980 24360 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.170.189" port: 32957 } has no permanent_uuid. Determining permanent_uuid...
Sep 23 20:14:50 dist-test-slave-th4g krb5kdc[24132](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194488, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.170.190@KRBTEST.COM
Sep 23 20:14:50 dist-test-slave-th4g krb5kdc[24132](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194490, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.170.188@KRBTEST.COM for kudu/127.1.170.189@KRBTEST.COM
I20260923 20:14:50.272302 24364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:14:50.256031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33572 (local address 127.1.170.190:39017)
0923 20:14:50.256189 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0923 20:14:50.256195 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:14:50.256211 (+ 16us) server_negotiation.cc:407] Connection header received
0923 20:14:50.256253 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:14:50.256258 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:14:50.256350 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:14:50.256474 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:14:50.257684 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:14:50.258739 (+ 1055us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:14:50.260095 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:14:50.260518 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:14:50.260617 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:14:50.265817 (+ 5200us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:14:50.265822 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:14:50.265825 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:14:50.265855 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:14:50.269357 (+ 3502us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:14:50.270176 (+ 819us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:14:50.270179 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:14:50.270182 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:14:50.270321 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:14:50.270795 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:14:50.270798 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:14:50.270800 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:14:50.271170 (+ 370us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:14:50.271451 (+ 281us) server_negotiation.cc:1035] Waiting for connection context
0923 20:14:50.271764 (+ 313us) server_negotiation.cc:299] Negotiation successful
0923 20:14:50.271988 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260923 20:14:50.272424 24365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:14:50.257667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.189:32957 (local address 127.0.0.1:39410)
0923 20:14:50.257801 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0923 20:14:50.257819 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0923 20:14:50.257973 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:14:50.258362 (+ 389us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:14:50.258366 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:14:50.258764 (+ 398us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:14:50.258996 (+ 232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:14:50.259008 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:14:50.259990 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:14:50.259997 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:14:50.261171 (+ 1174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:14:50.261182 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:14:50.261991 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:14:50.261998 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:14:50.262239 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:14:50.262802 (+ 563us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:14:50.262827 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:14:50.266024 (+ 3197us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:14:50.269704 (+ 3680us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:14:50.269709 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:14:50.269713 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:14:50.270252 (+ 539us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:14:50.270706 (+ 454us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:14:50.270715 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:14:50.270720 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:14:50.270874 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:14:50.271496 (+ 622us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:14:50.271501 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:14:50.271731 (+ 230us) client_negotiation.cc:769] Sending connection context
0923 20:14:50.271866 (+ 135us) client_negotiation.cc:240] Negotiation successful
0923 20:14:50.272140 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260923 20:14:50.272790 24290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:14:50.257801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39410 (local address 127.1.170.189:32957)
0923 20:14:50.258015 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0923 20:14:50.258021 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:14:50.258040 (+ 19us) server_negotiation.cc:407] Connection header received
0923 20:14:50.258103 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:14:50.258110 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:14:50.258194 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:14:50.258309 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:14:50.259173 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:14:50.259847 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:14:50.261330 (+ 1483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:14:50.261792 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:14:50.261949 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:14:50.266214 (+ 4265us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:14:50.266222 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:14:50.266228 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:14:50.266272 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:14:50.269575 (+ 3303us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:14:50.270363 (+ 788us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:14:50.270371 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:14:50.270375 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:14:50.270527 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:14:50.271001 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:14:50.271005 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:14:50.271008 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:14:50.271355 (+ 347us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:14:50.271495 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0923 20:14:50.272016 (+ 521us) server_negotiation.cc:299] Negotiation successful
0923 20:14:50.272433 (+ 417us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260923 20:14:50.273785 24360 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.170.188" port: 43729 } has no permanent_uuid. Determining permanent_uuid...
Sep 23 20:14:50 dist-test-slave-th4g krb5kdc[24132](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194490, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.170.188@KRBTEST.COM for kudu/127.1.170.188@KRBTEST.COM
Sep 23 20:14:50 dist-test-slave-th4g krb5kdc[24132](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194488, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.170.189@KRBTEST.COM
I20260923 20:14:50.287885 24363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:14:50.274554 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48822 (local address 127.1.170.188:43729)
0923 20:14:50.274693 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0923 20:14:50.274701 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 20:14:50.274864 (+ 163us) server_negotiation.cc:407] Connection header received
0923 20:14:50.275001 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:14:50.275005 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:14:50.275056 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:14:50.275125 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:14:50.276033 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:14:50.277071 (+ 1038us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:14:50.278747 (+ 1676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:14:50.279273 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:14:50.279431 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:14:50.282840 (+ 3409us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:14:50.282845 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:14:50.282848 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:14:50.282874 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:14:50.285224 (+ 2350us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:14:50.285990 (+ 766us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:14:50.285994 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:14:50.285996 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:14:50.286141 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:14:50.286581 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:14:50.286585 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:14:50.286587 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:14:50.286854 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:14:50.286966 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0923 20:14:50.287426 (+ 460us) server_negotiation.cc:299] Negotiation successful
0923 20:14:50.287620 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":48,"server-negotiator.queue_time_us":35}
I20260923 20:14:50.288214 24365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:14:50.274530 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.188:43729 (local address 127.0.0.1:48822)
0923 20:14:50.274778 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0923 20:14:50.274793 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0923 20:14:50.274904 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:14:50.275186 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:14:50.275191 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:14:50.275671 (+ 480us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:14:50.275903 (+ 232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:14:50.275916 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:14:50.277212 (+ 1296us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:14:50.277216 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:14:50.278608 (+ 1392us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:14:50.278624 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:14:50.279444 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:14:50.279449 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:14:50.279617 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:14:50.280217 (+ 600us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:14:50.280247 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:14:50.282651 (+ 2404us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:14:50.285359 (+ 2708us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:14:50.285367 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:14:50.285372 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:14:50.285835 (+ 463us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:14:50.286265 (+ 430us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:14:50.286298 (+ 33us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:14:50.286302 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:14:50.286462 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:14:50.286978 (+ 516us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:14:50.286985 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:14:50.287166 (+ 181us) client_negotiation.cc:769] Sending connection context
0923 20:14:50.287329 (+ 163us) client_negotiation.cc:240] Negotiation successful
0923 20:14:50.287825 (+ 496us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":165,"mutex_wait_us":115}
I20260923 20:14:50.291513 24290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:14:50.274076 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39412 (local address 127.1.170.189:32957)
0923 20:14:50.274214 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0923 20:14:50.274221 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:14:50.274238 (+ 17us) server_negotiation.cc:407] Connection header received
0923 20:14:50.274426 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:14:50.274432 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:14:50.274504 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:14:50.274595 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:14:50.276290 (+ 1695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:14:50.277310 (+ 1020us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:14:50.278992 (+ 1682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:14:50.279613 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:14:50.279716 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:14:50.285475 (+ 5759us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:14:50.285484 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:14:50.285489 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:14:50.285527 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:14:50.288868 (+ 3341us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:14:50.289622 (+ 754us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:14:50.289626 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:14:50.289628 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:14:50.289757 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:14:50.290179 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:14:50.290183 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:14:50.290185 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:14:50.290436 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:14:50.290715 (+ 279us) server_negotiation.cc:1035] Waiting for connection context
0923 20:14:50.290993 (+ 278us) server_negotiation.cc:299] Negotiation successful
0923 20:14:50.291214 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260923 20:14:50.293318 24360 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 04f672b75a424bfea249ca253ea6be46: Bootstrap starting.
I20260923 20:14:50.296262 1706 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 20:14:50.296868 24360 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 04f672b75a424bfea249ca253ea6be46: Neither blocks nor log segments found. Creating new log.
I20260923 20:14:50.297952 24360 log.cc:824] T 00000000000000000000000000000000 P 04f672b75a424bfea249ca253ea6be46: Log is configured to *not* fsync() on all Append() calls
I20260923 20:14:50.301209 24360 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 04f672b75a424bfea249ca253ea6be46: No bootstrap required, opened a new log
I20260923 20:14:50.299248 24187 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "405a4d054e444ce99c6a927df9098557"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33572
I20260923 20:14:50.312013 24360 raft_consensus.cc:348] T 00000000000000000000000000000000 P 04f672b75a424bfea249ca253ea6be46 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "405a4d054e444ce99c6a927df9098557" member_type: VOTER last_known_addr { host: "127.1.170.190" port: 39017 } } peers { permanent_uuid: "657c5e3e229e4b6b95203d69fae92453" member_type: VOTER last_known_addr { host: "127.1.170.189" port: 32957 } } peers { permanent_uuid: "04f672b75a424bfea249ca253ea6be46" member_type: VOTER last_known_addr { host: "127.1.170.188" port: 43729 } }
I20260923 20:14:50.312422 24360 raft_consensus.cc:729] T 00000000000000000000000000000000 P 04f672b75a424bfea249ca253ea6be46 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 04f672b75a424bfea249ca253ea6be46, State: Initialized, Role: FOLLOWER
I20260923 20:14:50.312935 24360 consensus_queue.cc:260] T 00000000000000000000000000000000 P 04f672b75a424bfea249ca253ea6be46 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "405a4d054e444ce99c6a927df9098557" member_type: VOTER last_known_addr { host: "127.1.170.190" port: 39017 } } peers { permanent_uuid: "657c5e3e229e4b6b95203d69fae92453" member_type: VOTER last_known_addr { host: "127.1.170.189" port: 32957 } } peers { permanent_uuid: "04f672b75a424bfea249ca253ea6be46" member_type: VOTER last_known_addr { host: "127.1.170.188" port: 43729 } }
I20260923 20:14:50.313825 24367 sys_catalog.cc:455] T 00000000000000000000000000000000 P 04f672b75a424bfea249ca253ea6be46 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "405a4d054e444ce99c6a927df9098557" member_type: VOTER last_known_addr { host: "127.1.170.190" port: 39017 } } peers { permanent_uuid: "657c5e3e229e4b6b95203d69fae92453" member_type: VOTER last_known_addr { host: "127.1.170.189" port: 32957 } } peers { permanent_uuid: "04f672b75a424bfea249ca253ea6be46" member_type: VOTER last_known_addr { host: "127.1.170.188" port: 43729 } } }
I20260923 20:14:50.314214 24367 sys_catalog.cc:458] T 00000000000000000000000000000000 P 04f672b75a424bfea249ca253ea6be46 [sys.catalog]: This master's current role is: FOLLOWER
I20260923 20:14:50.315085 24360 sys_catalog.cc:564] T 00000000000000000000000000000000 P 04f672b75a424bfea249ca253ea6be46 [sys.catalog]: configured and running, proceeding with master startup.
W20260923 20:14:50.325004 24378 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 04f672b75a424bfea249ca253ea6be46: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260923 20:14:50.325156 24378 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260923 20:14:50.322521 24293 master_runner.cc:186] Error getting master registration for 127.1.170.190:39017: OK, instance_id { permanent_uuid: "405a4d054e444ce99c6a927df9098557" instance_seqno: 1790194489202192 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260923 20:14:50.330040 24293 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260923 20:14:50.344609 24212 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.170.188" port: 43729 } attempt: 4
Sep 23 20:14:50 dist-test-slave-th4g krb5kdc[24132](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194489, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.170.190@KRBTEST.COM for kudu/127.1.170.188@KRBTEST.COM
I20260923 20:14:50.360327 24214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:14:50.345418 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.188:43729 (local address 127.0.0.1:48838)
0923 20:14:50.345614 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0923 20:14:50.345635 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0923 20:14:50.345776 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:14:50.346155 (+ 379us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:14:50.346161 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:14:50.346551 (+ 390us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:14:50.346721 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:14:50.346730 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:14:50.347781 (+ 1051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:14:50.347788 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:14:50.348734 (+ 946us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:14:50.348746 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:14:50.349563 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:14:50.349569 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:14:50.349773 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:14:50.350473 (+ 700us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:14:50.350495 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:14:50.353417 (+ 2922us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:14:50.357492 (+ 4075us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:14:50.357499 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:14:50.357504 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:14:50.358044 (+ 540us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:14:50.358437 (+ 393us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:14:50.358442 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:14:50.358447 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:14:50.358621 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:14:50.359252 (+ 631us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:14:50.359260 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:14:50.359430 (+ 170us) client_negotiation.cc:769] Sending connection context
0923 20:14:50.359587 (+ 157us) client_negotiation.cc:240] Negotiation successful
0923 20:14:50.359882 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":65}
I20260923 20:14:50.360365 24363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:14:50.345608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48838 (local address 127.1.170.188:43729)
0923 20:14:50.345803 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0923 20:14:50.345812 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 20:14:50.345837 (+ 25us) server_negotiation.cc:407] Connection header received
0923 20:14:50.345936 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:14:50.345941 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:14:50.346018 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:14:50.346109 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:14:50.346871 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:14:50.347623 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:14:50.348873 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:14:50.349407 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:14:50.349540 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:14:50.353657 (+ 4117us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:14:50.353668 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:14:50.353689 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:14:50.353731 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:14:50.357346 (+ 3615us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:14:50.358158 (+ 812us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:14:50.358163 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:14:50.358167 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:14:50.358299 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:14:50.358732 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:14:50.358737 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:14:50.358740 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:14:50.359106 (+ 366us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:14:50.359267 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0923 20:14:50.359696 (+ 429us) server_negotiation.cc:299] Negotiation successful
0923 20:14:50.359937 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260923 20:14:50.366300 24212 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 405a4d054e444ce99c6a927df9098557: Bootstrap starting.
I20260923 20:14:50.369418 24212 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 405a4d054e444ce99c6a927df9098557: Neither blocks nor log segments found. Creating new log.
I20260923 20:14:50.370517 24212 log.cc:824] T 00000000000000000000000000000000 P 405a4d054e444ce99c6a927df9098557: Log is configured to *not* fsync() on all Append() calls
I20260923 20:14:50.373167 24212 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 405a4d054e444ce99c6a927df9098557: No bootstrap required, opened a new log
I20260923 20:14:50.378876 24212 raft_consensus.cc:348] T 00000000000000000000000000000000 P 405a4d054e444ce99c6a927df9098557 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "405a4d054e444ce99c6a927df9098557" member_type: VOTER last_known_addr { host: "127.1.170.190" port: 39017 } } peers { permanent_uuid: "657c5e3e229e4b6b95203d69fae92453" member_type: VOTER last_known_addr { host: "127.1.170.189" port: 32957 } } peers { permanent_uuid: "04f672b75a424bfea249ca253ea6be46" member_type: VOTER last_known_addr { host: "127.1.170.188" port: 43729 } }
I20260923 20:14:50.379208 24212 raft_consensus.cc:729] T 00000000000000000000000000000000 P 405a4d054e444ce99c6a927df9098557 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 405a4d054e444ce99c6a927df9098557, State: Initialized, Role: FOLLOWER
I20260923 20:14:50.379650 24212 consensus_queue.cc:260] T 00000000000000000000000000000000 P 405a4d054e444ce99c6a927df9098557 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "405a4d054e444ce99c6a927df9098557" member_type: VOTER last_known_addr { host: "127.1.170.190" port: 39017 } } peers { permanent_uuid: "657c5e3e229e4b6b95203d69fae92453" member_type: VOTER last_known_addr { host: "127.1.170.189" port: 32957 } } peers { permanent_uuid: "04f672b75a424bfea249ca253ea6be46" member_type: VOTER last_known_addr { host: "127.1.170.188" port: 43729 } }
I20260923 20:14:50.386572 24381 sys_catalog.cc:455] T 00000000000000000000000000000000 P 405a4d054e444ce99c6a927df9098557 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "405a4d054e444ce99c6a927df9098557" member_type: VOTER last_known_addr { host: "127.1.170.190" port: 39017 } } peers { permanent_uuid: "657c5e3e229e4b6b95203d69fae92453" member_type: VOTER last_known_addr { host: "127.1.170.189" port: 32957 } } peers { permanent_uuid: "04f672b75a424bfea249ca253ea6be46" member_type: VOTER last_known_addr { host: "127.1.170.188" port: 43729 } } }
I20260923 20:14:50.386910 24381 sys_catalog.cc:458] T 00000000000000000000000000000000 P 405a4d054e444ce99c6a927df9098557 [sys.catalog]: This master's current role is: FOLLOWER
I20260923 20:14:50.388724 24212 sys_catalog.cc:564] T 00000000000000000000000000000000 P 405a4d054e444ce99c6a927df9098557 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 20:14:50.388794 24187 raft_consensus.cc:482] T 00000000000000000000000000000000 P 405a4d054e444ce99c6a927df9098557 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260923 20:14:50.388991 24187 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 405a4d054e444ce99c6a927df9098557 [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:14:50.393290 24187 raft_consensus.cc:504] T 00000000000000000000000000000000 P 405a4d054e444ce99c6a927df9098557 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "405a4d054e444ce99c6a927df9098557" member_type: VOTER last_known_addr { host: "127.1.170.190" port: 39017 } } peers { permanent_uuid: "657c5e3e229e4b6b95203d69fae92453" member_type: VOTER last_known_addr { host: "127.1.170.189" port: 32957 } } peers { permanent_uuid: "04f672b75a424bfea249ca253ea6be46" member_type: VOTER last_known_addr { host: "127.1.170.188" port: 43729 } }
I20260923 20:14:50.396234 24187 leader_election.cc:288] T 00000000000000000000000000000000 P 405a4d054e444ce99c6a927df9098557 [CANDIDATE]: Term 1 election: Requested vote from peers 657c5e3e229e4b6b95203d69fae92453 (127.1.170.189:32957), 04f672b75a424bfea249ca253ea6be46 (127.1.170.188:43729)
I20260923 20:14:50.396267 24335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "405a4d054e444ce99c6a927df9098557" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "04f672b75a424bfea249ca253ea6be46"
I20260923 20:14:50.396872 24335 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 04f672b75a424bfea249ca253ea6be46 [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:14:50.401543 24335 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 04f672b75a424bfea249ca253ea6be46 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 405a4d054e444ce99c6a927df9098557 in term 1.
I20260923 20:14:50.402413 24163 leader_election.cc:302] T 00000000000000000000000000000000 P 405a4d054e444ce99c6a927df9098557 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 04f672b75a424bfea249ca253ea6be46, 405a4d054e444ce99c6a927df9098557; no voters:
I20260923 20:14:50.402958 24381 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 405a4d054e444ce99c6a927df9098557 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 20:14:50.418006 24381 raft_consensus.cc:686] T 00000000000000000000000000000000 P 405a4d054e444ce99c6a927df9098557 [term 1 LEADER]: Becoming Leader. State: Replica: 405a4d054e444ce99c6a927df9098557, State: Running, Role: LEADER
I20260923 20:14:50.418660 24381 consensus_queue.cc:237] T 00000000000000000000000000000000 P 405a4d054e444ce99c6a927df9098557 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "405a4d054e444ce99c6a927df9098557" member_type: VOTER last_known_addr { host: "127.1.170.190" port: 39017 } } peers { permanent_uuid: "657c5e3e229e4b6b95203d69fae92453" member_type: VOTER last_known_addr { host: "127.1.170.189" port: 32957 } } peers { permanent_uuid: "04f672b75a424bfea249ca253ea6be46" member_type: VOTER last_known_addr { host: "127.1.170.188" port: 43729 } }
I20260923 20:14:50.423189 24286 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.170.188" port: 43729 } attempt: 5
I20260923 20:14:50.429688 24390 sys_catalog.cc:455] T 00000000000000000000000000000000 P 405a4d054e444ce99c6a927df9098557 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 405a4d054e444ce99c6a927df9098557. Latest consensus state: current_term: 1 leader_uuid: "405a4d054e444ce99c6a927df9098557" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "405a4d054e444ce99c6a927df9098557" member_type: VOTER last_known_addr { host: "127.1.170.190" port: 39017 } } peers { permanent_uuid: "657c5e3e229e4b6b95203d69fae92453" member_type: VOTER last_known_addr { host: "127.1.170.189" port: 32957 } } peers { permanent_uuid: "04f672b75a424bfea249ca253ea6be46" member_type: VOTER last_known_addr { host: "127.1.170.188" port: 43729 } } }
I20260923 20:14:50.430099 24390 sys_catalog.cc:458] T 00000000000000000000000000000000 P 405a4d054e444ce99c6a927df9098557 [sys.catalog]: This master's current role is: LEADER
I20260923 20:14:50.433861 24397 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
Sep 23 20:14:50 dist-test-slave-th4g krb5kdc[24132](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194489, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.170.189@KRBTEST.COM for kudu/127.1.170.188@KRBTEST.COM
W20260923 20:14:50.441351 24166 tablet.cc:2369] T 00000000000000000000000000000000 P 405a4d054e444ce99c6a927df9098557: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 20:14:50.454139 24289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:14:50.423999 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.188:43729 (local address 127.0.0.1:48854)
0923 20:14:50.424476 (+ 477us) negotiation.cc:107] Waiting for socket to connect
0923 20:14:50.424499 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0923 20:14:50.424641 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:14:50.425566 (+ 925us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:14:50.425574 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:14:50.426107 (+ 533us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:14:50.426320 (+ 213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:14:50.426331 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:14:50.427825 (+ 1494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:14:50.427833 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:14:50.429235 (+ 1402us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:14:50.429248 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:14:50.430819 (+ 1571us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:14:50.430828 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:14:50.431160 (+ 332us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:14:50.432040 (+ 880us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:14:50.432074 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:14:50.436728 (+ 4654us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:14:50.450278 (+ 13550us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:14:50.450284 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:14:50.450300 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:14:50.451041 (+ 741us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:14:50.451680 (+ 639us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:14:50.451687 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:14:50.451694 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:14:50.451895 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:14:50.452763 (+ 868us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:14:50.452772 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:14:50.452982 (+ 210us) client_negotiation.cc:769] Sending connection context
0923 20:14:50.453162 (+ 180us) client_negotiation.cc:240] Negotiation successful
0923 20:14:50.453567 (+ 405us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":329}
W20260923 20:14:50.454643 24314 tablet.cc:2369] T 00000000000000000000000000000000 P 04f672b75a424bfea249ca253ea6be46: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 20:14:50.455592 24363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:14:50.424964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48854 (local address 127.1.170.188:43729)
0923 20:14:50.425123 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0923 20:14:50.425130 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:14:50.425149 (+ 19us) server_negotiation.cc:407] Connection header received
0923 20:14:50.425210 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:14:50.425215 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:14:50.425285 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:14:50.425379 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:14:50.426488 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:14:50.427687 (+ 1199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:14:50.429387 (+ 1700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:14:50.430202 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:14:50.430400 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:14:50.437440 (+ 7040us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:14:50.437447 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:14:50.437451 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:14:50.437491 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:14:50.449988 (+ 12497us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:14:50.451266 (+ 1278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:14:50.451284 (+ 18us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:14:50.451300 (+ 16us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:14:50.451509 (+ 209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:14:50.452047 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:14:50.452053 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:14:50.452057 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:14:50.452498 (+ 441us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:14:50.452688 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0923 20:14:50.454721 (+ 2033us) server_negotiation.cc:299] Negotiation successful
0923 20:14:50.455097 (+ 376us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260923 20:14:50.470077 24286 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 657c5e3e229e4b6b95203d69fae92453: Bootstrap starting.
I20260923 20:14:50.474413 24398 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 20:14:50.477742 24286 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 657c5e3e229e4b6b95203d69fae92453: Neither blocks nor log segments found. Creating new log.
I20260923 20:14:50.440573 24145 master_runner.cc:186] Error getting master registration for 127.1.170.189:32957: OK, instance_id { permanent_uuid: "657c5e3e229e4b6b95203d69fae92453" instance_seqno: 1790194489693230 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260923 20:14:50.483673 24286 log.cc:824] T 00000000000000000000000000000000 P 657c5e3e229e4b6b95203d69fae92453: Log is configured to *not* fsync() on all Append() calls
I20260923 20:14:50.484988 24145 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260923 20:14:50.488039 24286 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 657c5e3e229e4b6b95203d69fae92453: No bootstrap required, opened a new log
I20260923 20:14:50.499054 24398 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 20:14:50.546319 24335 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 04f672b75a424bfea249ca253ea6be46 [term 1 FOLLOWER]: Refusing update from remote peer 405a4d054e444ce99c6a927df9098557: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 20:14:50.547394 24390 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 405a4d054e444ce99c6a927df9098557 [LEADER]: Connected to new peer: Peer: permanent_uuid: "04f672b75a424bfea249ca253ea6be46" member_type: VOTER last_known_addr { host: "127.1.170.188" port: 43729 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 20:14:50.560690 24364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:14:50.438140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33586 (local address 127.1.170.190:39017)
0923 20:14:50.438540 (+ 400us) server_negotiation.cc:206] Beginning negotiation
0923 20:14:50.438548 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 20:14:50.498278 (+ 59730us) server_negotiation.cc:407] Connection header received
0923 20:14:50.504235 (+ 5957us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:14:50.504244 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:14:50.504362 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:14:50.504456 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:14:50.507347 (+ 2891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:14:50.508718 (+ 1371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:14:50.513996 (+ 5278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:14:50.514859 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:14:50.514984 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:14:50.550049 (+ 35065us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:14:50.550057 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:14:50.550063 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:14:50.550132 (+ 69us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:14:50.553654 (+ 3522us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:14:50.557994 (+ 4340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:14:50.558003 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:14:50.558008 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:14:50.558193 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:14:50.558694 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:14:50.558701 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:14:50.558706 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:14:50.559121 (+ 415us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:14:50.559382 (+ 261us) server_negotiation.cc:1035] Waiting for connection context
0923 20:14:50.560018 (+ 636us) server_negotiation.cc:299] Negotiation successful
0923 20:14:50.560268 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":242}
I20260923 20:14:50.571094 24363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:14:50.498570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48858 (local address 127.1.170.188:43729)
0923 20:14:50.498774 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0923 20:14:50.498780 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:14:50.548232 (+ 49452us) server_negotiation.cc:407] Connection header received
0923 20:14:50.548428 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:14:50.548433 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:14:50.548514 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:14:50.548606 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:14:50.550646 (+ 2040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:14:50.551912 (+ 1266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:14:50.553964 (+ 2052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:14:50.554657 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:14:50.554745 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:14:50.560158 (+ 5413us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:14:50.560165 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:14:50.560169 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:14:50.560217 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:14:50.564066 (+ 3849us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:14:50.565009 (+ 943us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:14:50.565017 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:14:50.565022 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:14:50.565352 (+ 330us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:14:50.569105 (+ 3753us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:14:50.569111 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:14:50.569115 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:14:50.569504 (+ 389us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:14:50.569780 (+ 276us) server_negotiation.cc:1035] Waiting for connection context
0923 20:14:50.570370 (+ 590us) server_negotiation.cc:299] Negotiation successful
0923 20:14:50.570657 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260923 20:14:50.573722 24290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:14:50.438702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39420 (local address 127.1.170.189:32957)
0923 20:14:50.438902 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0923 20:14:50.438910 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 20:14:50.516226 (+ 77316us) server_negotiation.cc:407] Connection header received
0923 20:14:50.516604 (+ 378us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:14:50.516611 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:14:50.516716 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:14:50.516827 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:14:50.519818 (+ 2991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:14:50.521246 (+ 1428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:14:50.542194 (+ 20948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:14:50.543240 (+ 1046us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:14:50.543389 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:14:50.565041 (+ 21652us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:14:50.565050 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:14:50.565056 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:14:50.565107 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:14:50.568627 (+ 3520us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:14:50.570028 (+ 1401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:14:50.570036 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:14:50.570041 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:14:50.570222 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:14:50.570902 (+ 680us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:14:50.570910 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:14:50.570913 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:14:50.571588 (+ 675us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:14:50.571830 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0923 20:14:50.572782 (+ 952us) server_negotiation.cc:299] Negotiation successful
0923 20:14:50.573158 (+ 376us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260923 20:14:50.574901 24286 raft_consensus.cc:348] T 00000000000000000000000000000000 P 657c5e3e229e4b6b95203d69fae92453 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "405a4d054e444ce99c6a927df9098557" member_type: VOTER last_known_addr { host: "127.1.170.190" port: 39017 } } peers { permanent_uuid: "657c5e3e229e4b6b95203d69fae92453" member_type: VOTER last_known_addr { host: "127.1.170.189" port: 32957 } } peers { permanent_uuid: "04f672b75a424bfea249ca253ea6be46" member_type: VOTER last_known_addr { host: "127.1.170.188" port: 43729 } }
I20260923 20:14:50.575325 24286 raft_consensus.cc:729] T 00000000000000000000000000000000 P 657c5e3e229e4b6b95203d69fae92453 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 657c5e3e229e4b6b95203d69fae92453, State: Initialized, Role: FOLLOWER
I20260923 20:14:50.575855 24286 consensus_queue.cc:260] T 00000000000000000000000000000000 P 657c5e3e229e4b6b95203d69fae92453 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "405a4d054e444ce99c6a927df9098557" member_type: VOTER last_known_addr { host: "127.1.170.190" port: 39017 } } peers { permanent_uuid: "657c5e3e229e4b6b95203d69fae92453" member_type: VOTER last_known_addr { host: "127.1.170.189" port: 32957 } } peers { permanent_uuid: "04f672b75a424bfea249ca253ea6be46" member_type: VOTER last_known_addr { host: "127.1.170.188" port: 43729 } }
I20260923 20:14:50.577656 24407 sys_catalog.cc:455] T 00000000000000000000000000000000 P 657c5e3e229e4b6b95203d69fae92453 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "405a4d054e444ce99c6a927df9098557" member_type: VOTER last_known_addr { host: "127.1.170.190" port: 39017 } } peers { permanent_uuid: "657c5e3e229e4b6b95203d69fae92453" member_type: VOTER last_known_addr { host: "127.1.170.189" port: 32957 } } peers { permanent_uuid: "04f672b75a424bfea249ca253ea6be46" member_type: VOTER last_known_addr { host: "127.1.170.188" port: 43729 } } }
I20260923 20:14:50.578652 24407 sys_catalog.cc:458] T 00000000000000000000000000000000 P 657c5e3e229e4b6b95203d69fae92453 [sys.catalog]: This master's current role is: FOLLOWER
I20260923 20:14:50.579144 24286 sys_catalog.cc:564] T 00000000000000000000000000000000 P 657c5e3e229e4b6b95203d69fae92453 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 20:14:50.580367 24367 sys_catalog.cc:455] T 00000000000000000000000000000000 P 04f672b75a424bfea249ca253ea6be46 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 405a4d054e444ce99c6a927df9098557. Latest consensus state: current_term: 1 leader_uuid: "405a4d054e444ce99c6a927df9098557" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "405a4d054e444ce99c6a927df9098557" member_type: VOTER last_known_addr { host: "127.1.170.190" port: 39017 } } peers { permanent_uuid: "657c5e3e229e4b6b95203d69fae92453" member_type: VOTER last_known_addr { host: "127.1.170.189" port: 32957 } } peers { permanent_uuid: "04f672b75a424bfea249ca253ea6be46" member_type: VOTER last_known_addr { host: "127.1.170.188" port: 43729 } } }
I20260923 20:14:50.580716 24367 sys_catalog.cc:458] T 00000000000000000000000000000000 P 04f672b75a424bfea249ca253ea6be46 [sys.catalog]: This master's current role is: FOLLOWER
I20260923 20:14:50.579226 24261 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "405a4d054e444ce99c6a927df9098557" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "657c5e3e229e4b6b95203d69fae92453"
I20260923 20:14:50.579838 24260 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 657c5e3e229e4b6b95203d69fae92453 [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:14:50.584163 24381 sys_catalog.cc:455] T 00000000000000000000000000000000 P 405a4d054e444ce99c6a927df9098557 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "405a4d054e444ce99c6a927df9098557" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "405a4d054e444ce99c6a927df9098557" member_type: VOTER last_known_addr { host: "127.1.170.190" port: 39017 } } peers { permanent_uuid: "657c5e3e229e4b6b95203d69fae92453" member_type: VOTER last_known_addr { host: "127.1.170.189" port: 32957 } } peers { permanent_uuid: "04f672b75a424bfea249ca253ea6be46" member_type: VOTER last_known_addr { host: "127.1.170.188" port: 43729 } } }
I20260923 20:14:50.584468 24381 sys_catalog.cc:458] T 00000000000000000000000000000000 P 405a4d054e444ce99c6a927df9098557 [sys.catalog]: This master's current role is: LEADER
I20260923 20:14:50.586088 24260 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 657c5e3e229e4b6b95203d69fae92453 [term 1 FOLLOWER]: Refusing update from remote peer 405a4d054e444ce99c6a927df9098557: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 20:14:50.587282 24398 catalog_manager.cc:1347] Generated new cluster ID: eeadd5e450e5434c90dcfeeb8463c9f4
I20260923 20:14:50.587373 24398 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 20:14:50.589531 24261 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 657c5e3e229e4b6b95203d69fae92453 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 405a4d054e444ce99c6a927df9098557 in term 1.
I20260923 20:14:50.592334 24390 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 405a4d054e444ce99c6a927df9098557 [LEADER]: Connected to new peer: Peer: permanent_uuid: "657c5e3e229e4b6b95203d69fae92453" member_type: VOTER last_known_addr { host: "127.1.170.189" port: 32957 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 20:14:50.597847 24367 sys_catalog.cc:455] T 00000000000000000000000000000000 P 04f672b75a424bfea249ca253ea6be46 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "405a4d054e444ce99c6a927df9098557" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "405a4d054e444ce99c6a927df9098557" member_type: VOTER last_known_addr { host: "127.1.170.190" port: 39017 } } peers { permanent_uuid: "657c5e3e229e4b6b95203d69fae92453" member_type: VOTER last_known_addr { host: "127.1.170.189" port: 32957 } } peers { permanent_uuid: "04f672b75a424bfea249ca253ea6be46" member_type: VOTER last_known_addr { host: "127.1.170.188" port: 43729 } } }
I20260923 20:14:50.598168 24367 sys_catalog.cc:458] T 00000000000000000000000000000000 P 04f672b75a424bfea249ca253ea6be46 [sys.catalog]: This master's current role is: FOLLOWER
I20260923 20:14:50.608201 24398 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 20:14:50.610525 24398 catalog_manager.cc:1504] Loading token signing keys...
I20260923 20:14:50.628921 24398 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 405a4d054e444ce99c6a927df9098557: Generated new TSK 0
I20260923 20:14:50.630203 24398 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 20:14:50.635043 24407 sys_catalog.cc:455] T 00000000000000000000000000000000 P 657c5e3e229e4b6b95203d69fae92453 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 405a4d054e444ce99c6a927df9098557. Latest consensus state: current_term: 1 leader_uuid: "405a4d054e444ce99c6a927df9098557" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "405a4d054e444ce99c6a927df9098557" member_type: VOTER last_known_addr { host: "127.1.170.190" port: 39017 } } peers { permanent_uuid: "657c5e3e229e4b6b95203d69fae92453" member_type: VOTER last_known_addr { host: "127.1.170.189" port: 32957 } } peers { permanent_uuid: "04f672b75a424bfea249ca253ea6be46" member_type: VOTER last_known_addr { host: "127.1.170.188" port: 43729 } } }
I20260923 20:14:50.635126 24416 sys_catalog.cc:455] T 00000000000000000000000000000000 P 657c5e3e229e4b6b95203d69fae92453 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "405a4d054e444ce99c6a927df9098557" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "405a4d054e444ce99c6a927df9098557" member_type: VOTER last_known_addr { host: "127.1.170.190" port: 39017 } } peers { permanent_uuid: "657c5e3e229e4b6b95203d69fae92453" member_type: VOTER last_known_addr { host: "127.1.170.189" port: 32957 } } peers { permanent_uuid: "04f672b75a424bfea249ca253ea6be46" member_type: VOTER last_known_addr { host: "127.1.170.188" port: 43729 } } }
I20260923 20:14:50.635509 24416 sys_catalog.cc:458] T 00000000000000000000000000000000 P 657c5e3e229e4b6b95203d69fae92453 [sys.catalog]: This master's current role is: FOLLOWER
I20260923 20:14:50.635409 24407 sys_catalog.cc:458] T 00000000000000000000000000000000 P 657c5e3e229e4b6b95203d69fae92453 [sys.catalog]: This master's current role is: FOLLOWER
I20260923 20:14:50.636842 24390 sys_catalog.cc:455] T 00000000000000000000000000000000 P 405a4d054e444ce99c6a927df9098557 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "405a4d054e444ce99c6a927df9098557" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "405a4d054e444ce99c6a927df9098557" member_type: VOTER last_known_addr { host: "127.1.170.190" port: 39017 } } peers { permanent_uuid: "657c5e3e229e4b6b95203d69fae92453" member_type: VOTER last_known_addr { host: "127.1.170.189" port: 32957 } } peers { permanent_uuid: "04f672b75a424bfea249ca253ea6be46" member_type: VOTER last_known_addr { host: "127.1.170.188" port: 43729 } } }
I20260923 20:14:50.637091 24390 sys_catalog.cc:458] T 00000000000000000000000000000000 P 405a4d054e444ce99c6a927df9098557 [sys.catalog]: This master's current role is: LEADER
I20260923 20:14:50.669196 24427 catalog_manager.cc:1259] Loaded cluster ID: eeadd5e450e5434c90dcfeeb8463c9f4
I20260923 20:14:50.669373 24427 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 657c5e3e229e4b6b95203d69fae92453: loading cluster ID for follower catalog manager: success
I20260923 20:14:50.673123 24427 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 657c5e3e229e4b6b95203d69fae92453: acquiring CA information for follower catalog manager: success
I20260923 20:14:50.675874 24427 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 657c5e3e229e4b6b95203d69fae92453: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260923 20:14:50.756374 1706 mini_kdc.cc:348] Time spent kdestroy: real 0.113s user 0.007s sys 0.015s
W20260923 20:14:50.811254 24364 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 20:14:50.797612 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33600 (local address 127.1.170.190:39017)
0923 20:14:50.797833 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0923 20:14:50.797840 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:14:50.807974 (+ 10134us) server_negotiation.cc:407] Connection header received
0923 20:14:50.809464 (+ 1490us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:14:50.809471 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:14:50.809565 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:14:50.809646 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:14:50.810997 (+ 1351us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33600: BlockingRecv error: recv got EOF from 127.0.0.1:33600 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
W20260923 20:14:50.812297 24290 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 20:14:50.803659 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39430 (local address 127.1.170.189:32957)
0923 20:14:50.803871 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0923 20:14:50.803878 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:14:50.810475 (+ 6597us) server_negotiation.cc:407] Connection header received
0923 20:14:50.810772 (+ 297us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:14:50.810778 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:14:50.810896 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:14:50.810980 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:14:50.812064 (+ 1084us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39430: BlockingRecv error: recv got EOF from 127.0.0.1:39430 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
W20260923 20:14:50.814688 24363 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 20:14:50.808570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48874 (local address 127.1.170.188:43729)
0923 20:14:50.808775 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0923 20:14:50.808781 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:14:50.813000 (+ 4219us) server_negotiation.cc:407] Connection header received
0923 20:14:50.813266 (+ 266us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:14:50.813273 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:14:50.813364 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:14:50.813445 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:14:50.814415 (+ 970us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48874: BlockingRecv error: recv got EOF from 127.0.0.1:48874 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
I20260923 20:14:50.880849 24290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:14:50.860888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39436 (local address 127.1.170.189:32957)
0923 20:14:50.861108 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0923 20:14:50.861116 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 20:14:50.872759 (+ 11643us) server_negotiation.cc:407] Connection header received
0923 20:14:50.873121 (+ 362us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:14:50.873129 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:14:50.873356 (+ 227us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:14:50.873508 (+ 152us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 20:14:50.874305 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:14:50.875470 (+ 1165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:14:50.877524 (+ 2054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:14:50.878220 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:14:50.878319 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:14:50.879273 (+ 954us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 20:14:50.880007 (+ 734us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 20:14:50.880128 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0923 20:14:50.880372 (+ 244us) server_negotiation.cc:299] Negotiation successful
0923 20:14:50.880475 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
W20260923 20:14:50.936581 24363 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 20:14:50.919180 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48888 (local address 127.1.170.188:43729)
0923 20:14:50.919370 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0923 20:14:50.919376 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:14:50.934581 (+ 15205us) server_negotiation.cc:407] Connection header received
0923 20:14:50.934851 (+ 270us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:14:50.934858 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:14:50.935019 (+ 161us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:14:50.935137 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:14:50.936284 (+ 1147us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48888: BlockingRecv error: recv got EOF from 127.0.0.1:48888 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
I20260923 20:14:50.969102 24290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:14:50.960055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39448 (local address 127.1.170.189:32957)
0923 20:14:50.960284 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0923 20:14:50.960291 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:14:50.962508 (+ 2217us) server_negotiation.cc:407] Connection header received
0923 20:14:50.962725 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:14:50.962731 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:14:50.962867 (+ 136us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:14:50.962973 (+ 106us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 20:14:50.963461 (+ 488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:14:50.964724 (+ 1263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:14:50.966927 (+ 2203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:14:50.967602 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:14:50.967739 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:14:50.968197 (+ 458us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 20:14:50.968395 (+ 198us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 20:14:50.968497 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0923 20:14:50.968660 (+ 163us) server_negotiation.cc:299] Negotiation successful
0923 20:14:50.968749 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
W20260923 20:14:50.988031 24363 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 20:14:50.982354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48900 (local address 127.1.170.188:43729)
0923 20:14:50.982560 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0923 20:14:50.982566 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:14:50.986491 (+ 3925us) server_negotiation.cc:407] Connection header received
0923 20:14:50.986696 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:14:50.986702 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:14:50.986810 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:14:50.986923 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:14:50.987802 (+ 879us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48900: BlockingRecv error: recv got EOF from 127.0.0.1:48900 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
I20260923 20:14:51.000761 24290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:14:50.993410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39464 (local address 127.1.170.189:32957)
0923 20:14:50.993601 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0923 20:14:50.993608 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:14:50.994183 (+ 575us) server_negotiation.cc:407] Connection header received
0923 20:14:50.994421 (+ 238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:14:50.994427 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:14:50.994518 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:14:50.994626 (+ 108us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 20:14:50.995126 (+ 500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:14:50.996265 (+ 1139us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:14:50.998283 (+ 2018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:14:50.998971 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:14:50.999097 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:14:50.999607 (+ 510us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 20:14:50.999759 (+ 152us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 20:14:51.000011 (+ 252us) server_negotiation.cc:1035] Waiting for connection context
0923 20:14:51.000308 (+ 297us) server_negotiation.cc:299] Negotiation successful
0923 20:14:51.000397 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
W20260923 20:14:51.014341 24363 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 20:14:51.011947 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48914 (local address 127.1.170.188:43729)
0923 20:14:51.012138 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0923 20:14:51.012144 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:14:51.012683 (+ 539us) server_negotiation.cc:407] Connection header received
0923 20:14:51.012939 (+ 256us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:14:51.012945 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:14:51.013046 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:14:51.013159 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:14:51.014094 (+ 935us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48914: BlockingRecv error: recv got EOF from 127.0.0.1:48914 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
I20260923 20:14:51.028960 24290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:14:51.022093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39470 (local address 127.1.170.189:32957)
0923 20:14:51.022276 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0923 20:14:51.022282 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:14:51.022801 (+ 519us) server_negotiation.cc:407] Connection header received
0923 20:14:51.022993 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:14:51.022998 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:14:51.023096 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:14:51.023216 (+ 120us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 20:14:51.023682 (+ 466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:14:51.024889 (+ 1207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:14:51.026857 (+ 1968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:14:51.027553 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:14:51.027708 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:14:51.028155 (+ 447us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 20:14:51.028296 (+ 141us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 20:14:51.028394 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0923 20:14:51.028549 (+ 155us) server_negotiation.cc:299] Negotiation successful
0923 20:14:51.028643 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
W20260923 20:14:51.042532 24363 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 20:14:51.039983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48928 (local address 127.1.170.188:43729)
0923 20:14:51.040275 (+ 292us) server_negotiation.cc:206] Beginning negotiation
0923 20:14:51.040283 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 20:14:51.040909 (+ 626us) server_negotiation.cc:407] Connection header received
0923 20:14:51.041118 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:14:51.041126 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:14:51.041242 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:14:51.041354 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:14:51.042297 (+ 943us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48928: BlockingRecv error: recv got EOF from 127.0.0.1:48928 (error 108)
Metrics: {"server-negotiator.queue_time_us":162}
I20260923 20:14:51.065006 24290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:14:51.057009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39478 (local address 127.1.170.189:32957)
0923 20:14:51.057211 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0923 20:14:51.057218 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:14:51.057857 (+ 639us) server_negotiation.cc:407] Connection header received
0923 20:14:51.058240 (+ 383us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:14:51.058248 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:14:51.058365 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:14:51.058487 (+ 122us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 20:14:51.059009 (+ 522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:14:51.060348 (+ 1339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:14:51.062438 (+ 2090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:14:51.063292 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:14:51.063446 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:14:51.063930 (+ 484us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 20:14:51.064120 (+ 190us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 20:14:51.064217 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0923 20:14:51.064523 (+ 306us) server_negotiation.cc:299] Negotiation successful
0923 20:14:51.064621 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
W20260923 20:14:51.082074 24363 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 20:14:51.079580 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48940 (local address 127.1.170.188:43729)
0923 20:14:51.079806 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0923 20:14:51.079813 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:14:51.080369 (+ 556us) server_negotiation.cc:407] Connection header received
0923 20:14:51.080573 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:14:51.080582 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:14:51.080692 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:14:51.080826 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:14:51.081778 (+ 952us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48940: BlockingRecv error: recv got EOF from 127.0.0.1:48940 (error 108)
Metrics: {"server-negotiator.queue_time_us":83}
I20260923 20:14:51.121171 24290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:14:51.113425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39480 (local address 127.1.170.189:32957)
0923 20:14:51.113619 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0923 20:14:51.113627 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 20:14:51.114215 (+ 588us) server_negotiation.cc:407] Connection header received
0923 20:14:51.114421 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:14:51.114428 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:14:51.114524 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:14:51.114631 (+ 107us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 20:14:51.115174 (+ 543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:14:51.116454 (+ 1280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:14:51.118663 (+ 2209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:14:51.119442 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:14:51.119613 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:14:51.120192 (+ 579us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 20:14:51.120353 (+ 161us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 20:14:51.120430 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0923 20:14:51.120697 (+ 267us) server_negotiation.cc:299] Negotiation successful
0923 20:14:51.120816 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
W20260923 20:14:51.132211 24363 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 20:14:51.129149 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48948 (local address 127.1.170.188:43729)
0923 20:14:51.129356 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0923 20:14:51.129365 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 20:14:51.130596 (+ 1231us) server_negotiation.cc:407] Connection header received
0923 20:14:51.130817 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:14:51.130824 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:14:51.130923 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:14:51.131029 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:14:51.131978 (+ 949us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48948: BlockingRecv error: recv got EOF from 127.0.0.1:48948 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
I20260923 20:14:51.173635 24290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:14:51.168017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39492 (local address 127.1.170.189:32957)
0923 20:14:51.168255 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0923 20:14:51.168264 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 20:14:51.168718 (+ 454us) server_negotiation.cc:407] Connection header received
0923 20:14:51.168937 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:14:51.168943 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:14:51.169059 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:14:51.169178 (+ 119us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 20:14:51.169609 (+ 431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:14:51.170463 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:14:51.171812 (+ 1349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:14:51.172309 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:14:51.172455 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:14:51.172872 (+ 417us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 20:14:51.172981 (+ 109us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 20:14:51.173091 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0923 20:14:51.173243 (+ 152us) server_negotiation.cc:299] Negotiation successful
0923 20:14:51.173334 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
W20260923 20:14:51.182880 24363 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 20:14:51.180914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48950 (local address 127.1.170.188:43729)
0923 20:14:51.181147 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0923 20:14:51.181155 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 20:14:51.181409 (+ 254us) server_negotiation.cc:407] Connection header received
0923 20:14:51.181656 (+ 247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:14:51.181662 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:14:51.181746 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:14:51.181852 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:14:51.182632 (+ 780us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48950: BlockingRecv error: recv got EOF from 127.0.0.1:48950 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
I20260923 20:14:51.255723 24290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:14:51.250871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39500 (local address 127.1.170.189:32957)
0923 20:14:51.251150 (+ 279us) server_negotiation.cc:206] Beginning negotiation
0923 20:14:51.251157 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:14:51.251357 (+ 200us) server_negotiation.cc:407] Connection header received
0923 20:14:51.251514 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:14:51.251518 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:14:51.251591 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:14:51.251673 (+ 82us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 20:14:51.252040 (+ 367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:14:51.252810 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:14:51.254165 (+ 1355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:14:51.254561 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:14:51.254657 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:14:51.255024 (+ 367us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 20:14:51.255128 (+ 104us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 20:14:51.255195 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0923 20:14:51.255327 (+ 132us) server_negotiation.cc:299] Negotiation successful
0923 20:14:51.255394 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99}
W20260923 20:14:51.263906 24363 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 20:14:51.261441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48962 (local address 127.1.170.188:43729)
0923 20:14:51.261697 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0923 20:14:51.261704 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:14:51.261961 (+ 257us) server_negotiation.cc:407] Connection header received
0923 20:14:51.262114 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:14:51.262118 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:14:51.262218 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:14:51.262311 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:14:51.263682 (+ 1371us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48962: BlockingRecv error: recv got EOF from 127.0.0.1:48962 (error 108)
Metrics: {"server-negotiator.queue_time_us":95}
I20260923 20:14:51.327324 24378 catalog_manager.cc:1259] Loaded cluster ID: eeadd5e450e5434c90dcfeeb8463c9f4
I20260923 20:14:51.327423 24378 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 04f672b75a424bfea249ca253ea6be46: loading cluster ID for follower catalog manager: success
I20260923 20:14:51.329864 24378 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 04f672b75a424bfea249ca253ea6be46: acquiring CA information for follower catalog manager: success
I20260923 20:14:51.331670 24378 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 04f672b75a424bfea249ca253ea6be46: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260923 20:14:51.402052 24290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:14:51.395885 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39514 (local address 127.1.170.189:32957)
0923 20:14:51.396150 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0923 20:14:51.396161 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0923 20:14:51.396620 (+ 459us) server_negotiation.cc:407] Connection header received
0923 20:14:51.396882 (+ 262us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:14:51.396887 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:14:51.396962 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:14:51.397084 (+ 122us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 20:14:51.397598 (+ 514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:14:51.398447 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:14:51.400060 (+ 1613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:14:51.400624 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:14:51.400785 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:14:51.401216 (+ 431us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 20:14:51.401350 (+ 134us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 20:14:51.401462 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0923 20:14:51.401623 (+ 161us) server_negotiation.cc:299] Negotiation successful
0923 20:14:51.401714 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101}
I20260923 20:14:51.415196 24363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:14:51.408169 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48976 (local address 127.1.170.188:43729)
0923 20:14:51.408441 (+ 272us) server_negotiation.cc:206] Beginning negotiation
0923 20:14:51.408448 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:14:51.408647 (+ 199us) server_negotiation.cc:407] Connection header received
0923 20:14:51.408929 (+ 282us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:14:51.408934 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:14:51.409094 (+ 160us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:14:51.409256 (+ 162us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 20:14:51.409674 (+ 418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:14:51.410459 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:14:51.412258 (+ 1799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:14:51.412929 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:14:51.413273 (+ 344us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:14:51.413866 (+ 593us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 20:14:51.414499 (+ 633us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 20:14:51.414672 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0923 20:14:51.414778 (+ 106us) server_negotiation.cc:299] Negotiation successful
0923 20:14:51.414909 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260923 20:14:51.430037 24290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:14:51.422458 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39518 (local address 127.1.170.189:32957)
0923 20:14:51.422733 (+ 275us) server_negotiation.cc:206] Beginning negotiation
0923 20:14:51.422744 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0923 20:14:51.423380 (+ 636us) server_negotiation.cc:407] Connection header received
0923 20:14:51.423713 (+ 333us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:14:51.423721 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:14:51.423841 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:14:51.423910 (+ 69us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 20:14:51.424718 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:14:51.425865 (+ 1147us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:14:51.427717 (+ 1852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:14:51.428528 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:14:51.428766 (+ 238us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:14:51.429196 (+ 430us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 20:14:51.429342 (+ 146us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 20:14:51.429451 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0923 20:14:51.429604 (+ 153us) server_negotiation.cc:299] Negotiation successful
0923 20:14:51.429695 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260923 20:14:51.431378 24363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:14:51.423959 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48992 (local address 127.1.170.188:43729)
0923 20:14:51.424159 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0923 20:14:51.424165 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:14:51.424965 (+ 800us) server_negotiation.cc:407] Connection header received
0923 20:14:51.425199 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:14:51.425205 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:14:51.425343 (+ 138us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:14:51.425445 (+ 102us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 20:14:51.425997 (+ 552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:14:51.427005 (+ 1008us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:14:51.429323 (+ 2318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:14:51.430036 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:14:51.430158 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:14:51.430623 (+ 465us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 20:14:51.430727 (+ 104us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 20:14:51.430836 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0923 20:14:51.431014 (+ 178us) server_negotiation.cc:299] Negotiation successful
0923 20:14:51.431091 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260923 20:14:51.431372 24533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:14:51.422359 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33604 (local address 127.1.170.190:39017)
0923 20:14:51.424023 (+ 1664us) server_negotiation.cc:206] Beginning negotiation
0923 20:14:51.424031 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 20:14:51.424050 (+ 19us) server_negotiation.cc:407] Connection header received
0923 20:14:51.424183 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:14:51.424191 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:14:51.424424 (+ 233us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:14:51.424615 (+ 191us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 20:14:51.425405 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:14:51.426531 (+ 1126us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:14:51.428462 (+ 1931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:14:51.429062 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:14:51.429193 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:14:51.429850 (+ 657us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 20:14:51.430682 (+ 832us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 20:14:51.430793 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0923 20:14:51.430951 (+ 158us) server_negotiation.cc:299] Negotiation successful
0923 20:14:51.431039 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1542,"thread_start_us":1433,"threads_started":1}
I20260923 20:14:51.437265 1706 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskqUuku6/build/asan/bin/kudu pid 24145
I20260923 20:14:51.438562 24207 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:14:51.551797 1706 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqUuku6/build/asan/bin/kudu with pid 24145
W20260923 20:14:51.573017 24310 connection.cc:535] server connection from 127.0.0.1:48838 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260923 20:14:51.573016 24238 connection.cc:535] server connection from 127.0.0.1:39402 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260923 20:14:51.573534 1706 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskqUuku6/build/asan/bin/kudu pid 24219
I20260923 20:14:51.574754 24281 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:14:51.689738 1706 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqUuku6/build/asan/bin/kudu with pid 24219
I20260923 20:14:51.710330 1706 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskqUuku6/build/asan/bin/kudu pid 24293
I20260923 20:14:51.711813 24355 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:14:51.824203 1706 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqUuku6/build/asan/bin/kudu with pid 24293
2026-09-23T20:14:51Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (4484 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790194121577027-1706-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 20:14:51 dist-test-slave-th4g krb5kdc[24541](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 20:14:51 dist-test-slave-th4g krb5kdc[24541](info): set up 2 sockets
Sep 23 20:14:51 dist-test-slave-th4g krb5kdc[24541](info): commencing operation
krb5kdc: starting...
W20260923 20:14:52.958655 1706 mini_kdc.cc:121] Time spent starting KDC: real 1.092s user 0.006s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790194121577027-1706-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790194121577027-1706-0/krb5kdc/test-user.keytab.
Sep 23 20:14:53 dist-test-slave-th4g krb5kdc[24541](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194493, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T20:14:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T20:14:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.170.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.170.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.170.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790194121577027-1706-0/krb5kdc/kudu_127.1.170.190.keytab.
Entry for principal kudu/127.1.170.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790194121577027-1706-0/krb5kdc/kudu_127.1.170.190.keytab.
I20260923 20:14:53.222167 1706 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqUuku6/build/asan/bin/kudu
/tmp/dist-test-taskqUuku6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790194121577027-1706-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790194121577027-1706-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790194121577027-1706-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790194121577027-1706-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.170.190:33757
--webserver_interface=127.1.170.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790194121577027-1706-0/krb5kdc/kudu_127.1.170.190.keytab
--principal=kudu/127.1.170.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.170.148:34397
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.170.190:33757,127.1.170.189:34745,127.1.170.188:45387
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790194121577027-1706-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790194121577027-1706-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790194121577027-1706-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:14:53.557301 24554 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260923 20:14:53.557670 24554 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:14:53.557818 24554 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:14:53.557974 24554 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:14:53.565356 24554 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 20:14:53.565445 24554 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:14:53.565538 24554 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:14:53.565616 24554 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 20:14:53.565737 24554 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 20:14:53.575714 24554 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.170.148:34397
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790194121577027-1706-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790194121577027-1706-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.170.190:33757,127.1.170.189:34745,127.1.170.188:45387
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790194121577027-1706-0/krb5kdc/kudu_127.1.170.190.keytab
--principal=kudu/127.1.170.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.170.190:33757
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790194121577027-1706-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.170.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790194121577027-1706-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:14:53.577102 24554 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:14:53.578459 24554 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:14:53.592083 24559 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:14:53.591949 24560 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:14:53.592199 24562 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:14:53.593233 24554 server_base.cc:1034] running on GCE node
I20260923 20:14:53.594069 24554 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:14:53.595475 24554 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:14:53.596726 24554 hybrid_clock.cc:648] HybridClock initialized: now 1790194493596653 us; error 45 us; skew 500 ppm
I20260923 20:14:53.597126 24554 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:14:53 dist-test-slave-th4g krb5kdc[24541](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194493, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.170.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:14:53.600235 24554 init.cc:377] Logged in from keytab as kudu/127.1.170.190@KRBTEST.COM (short username kudu)
I20260923 20:14:53.601683 24554 webserver.cc:460] Webserver started at http://127.1.170.190:36873/ using document root <none> and password file <none>
I20260923 20:14:53.602423 24554 fs_manager.cc:359] Metadata directory not provided
I20260923 20:14:53.602562 24554 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:14:53.602900 24554 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:14:53.605806 24554 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790194121577027-1706-0/minicluster-data/master-0/data/instance:
uuid: "3d45e0d3fb0447afbff64afec99359d1"
format_stamp: "Formatted at 2026-09-23 20:14:53 on dist-test-slave-th4g"
server_key: "f83887f9ba081cd362041c0ad82bf2a7"
server_key_iv: "424b4b6a75151c800ef26bc00ff7695a"
server_key_version: "encryptionkey@0"
I20260923 20:14:53.606681 24554 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790194121577027-1706-0/minicluster-data/master-0/wal/instance:
uuid: "3d45e0d3fb0447afbff64afec99359d1"
format_stamp: "Formatted at 2026-09-23 20:14:53 on dist-test-slave-th4g"
server_key: "f83887f9ba081cd362041c0ad82bf2a7"
server_key_iv: "424b4b6a75151c800ef26bc00ff7695a"
server_key_version: "encryptionkey@0"
I20260923 20:14:53.611871 24554 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260923 20:14:53.615263 24569 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:14:53.616698 24554 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260923 20:14:53.616890 24554 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790194121577027-1706-0/minicluster-data/master-0/data,/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790194121577027-1706-0/minicluster-data/master-0/wal
uuid: "3d45e0d3fb0447afbff64afec99359d1"
format_stamp: "Formatted at 2026-09-23 20:14:53 on dist-test-slave-th4g"
server_key: "f83887f9ba081cd362041c0ad82bf2a7"
server_key_iv: "424b4b6a75151c800ef26bc00ff7695a"
server_key_version: "encryptionkey@0"
I20260923 20:14:53.617136 24554 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790194121577027-1706-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790194121577027-1706-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790194121577027-1706-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:14:53.640750 24554 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:14:53.644917 24554 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:14:53.645198 24554 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:14:53.663961 24554 rpc_server.cc:307] RPC server started. Bound to: 127.1.170.190:33757
I20260923 20:14:53.663980 24620 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.170.190:33757 every 8 connection(s)
I20260923 20:14:53.665354 24554 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790194121577027-1706-0/minicluster-data/master-0/data/info.pb
I20260923 20:14:53.667128 1706 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqUuku6/build/asan/bin/kudu as pid 24554
I20260923 20:14:53.667321 1706 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790194121577027-1706-0/minicluster-data/master-0/wal/instance
I20260923 20:14:53.667757 1706 external_mini_cluster.cc:1428] Setting key d212add3902236f9482e3620f201d88d
I20260923 20:14:53.669153 24621 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:14:53.675702 24621 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.170.190" port: 33757 } has no permanent_uuid. Determining permanent_uuid...
Sep 23 20:14:53 dist-test-slave-th4g krb5kdc[24541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194493, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.170.190@KRBTEST.COM for kudu/127.1.170.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.1.170.189@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.170.189@KRBTEST.COM" created.
I20260923 20:14:53.701936 24624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:14:53.679615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.190:33757 (local address 127.0.0.1:58108)
0923 20:14:53.680535 (+ 920us) negotiation.cc:107] Waiting for socket to connect
0923 20:14:53.680589 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0923 20:14:53.682811 (+ 2222us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:14:53.683599 (+ 788us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:14:53.683605 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:14:53.684452 (+ 847us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:14:53.684803 (+ 351us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:14:53.684822 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:14:53.686593 (+ 1771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:14:53.686602 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:14:53.688585 (+ 1983us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:14:53.688604 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:14:53.689834 (+ 1230us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:14:53.689845 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:14:53.690109 (+ 264us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:14:53.690893 (+ 784us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:14:53.690925 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:14:53.693998 (+ 3073us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:14:53.697991 (+ 3993us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:14:53.698001 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:14:53.698010 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:14:53.698409 (+ 399us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:14:53.698889 (+ 480us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:14:53.698892 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:14:53.698894 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:14:53.699037 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:14:53.700015 (+ 978us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:14:53.700021 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:14:53.700214 (+ 193us) client_negotiation.cc:769] Sending connection context
0923 20:14:53.700427 (+ 213us) client_negotiation.cc:240] Negotiation successful
0923 20:14:53.700826 (+ 399us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":650,"thread_start_us":408,"threads_started":1}
I20260923 20:14:53.701938 24625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:14:53.679620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58108 (local address 127.1.170.190:33757)
0923 20:14:53.682410 (+ 2790us) server_negotiation.cc:206] Beginning negotiation
0923 20:14:53.682418 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 20:14:53.682471 (+ 53us) server_negotiation.cc:407] Connection header received
0923 20:14:53.683158 (+ 687us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:14:53.683163 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:14:53.683388 (+ 225us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:14:53.683590 (+ 202us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:14:53.685022 (+ 1432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:14:53.686398 (+ 1376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:14:53.688806 (+ 2408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:14:53.689515 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:14:53.689683 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:14:53.694239 (+ 4556us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:14:53.694247 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:14:53.694260 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:14:53.694301 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:14:53.697779 (+ 3478us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:14:53.698551 (+ 772us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:14:53.698561 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:14:53.698567 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:14:53.698775 (+ 208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:14:53.699148 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:14:53.699155 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:14:53.699161 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:14:53.699873 (+ 712us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:14:53.700079 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0923 20:14:53.700603 (+ 524us) server_negotiation.cc:299] Negotiation successful
0923 20:14:53.701029 (+ 426us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":103,"server-negotiator.queue_time_us":2583,"thread_start_us":2303,"threads_started":1}
I20260923 20:14:53.704983 24621 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.170.189" port: 34745 } has no permanent_uuid. Determining permanent_uuid...
I20260923 20:14:53.706137 24624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:14:53.705674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.189:34745 (local address 127.0.0.1:51750)
0923 20:14:53.705855 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0923 20:14:53.706026 (+ 171us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.170.189:34745: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260923 20:14:53.706435 24572 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.170.189:34745: connect: Connection refused (error 111)
I20260923 20:14:53.708114 24624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:14:53.707648 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.189:34745 (local address 127.0.0.1:51764)
0923 20:14:53.707835 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0923 20:14:53.707938 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.170.189:34745: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260923 20:14:53.708339 24621 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.170.189:34745: Network error: Client connection negotiation failed: client connection to 127.1.170.189:34745: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.170.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790194121577027-1706-0/krb5kdc/kudu_127.1.170.189.keytab.
Entry for principal kudu/127.1.170.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790194121577027-1706-0/krb5kdc/kudu_127.1.170.189.keytab.
I20260923 20:14:53.729645 1706 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqUuku6/build/asan/bin/kudu
/tmp/dist-test-taskqUuku6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790194121577027-1706-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790194121577027-1706-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790194121577027-1706-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790194121577027-1706-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.170.189:34745
--webserver_interface=127.1.170.189
--webserver_port=0
--keytab_file=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790194121577027-1706-0/krb5kdc/kudu_127.1.170.189.keytab
--principal=kudu/127.1.170.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.170.148:34397
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.170.190:33757,127.1.170.189:34745,127.1.170.188:45387
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790194121577027-1706-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790194121577027-1706-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790194121577027-1706-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260923 20:14:53.757735 24621 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.170.189" port: 34745 } attempt: 1
I20260923 20:14:53.759191 24624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:14:53.758573 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.189:34745 (local address 127.0.0.1:51774)
0923 20:14:53.758805 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0923 20:14:53.758989 (+ 184us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.170.189:34745: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260923 20:14:53.760488 24624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:14:53.760051 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.189:34745 (local address 127.0.0.1:51790)
0923 20:14:53.760262 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0923 20:14:53.760342 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.170.189:34745: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108}
W20260923 20:14:53.760710 24621 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.170.189:34745: Network error: Client connection negotiation failed: client connection to 127.1.170.189:34745: connect: Connection refused (error 111)
I20260923 20:14:53.828980 24621 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.170.189" port: 34745 } attempt: 2
I20260923 20:14:53.830341 24624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:14:53.829770 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.189:34745 (local address 127.0.0.1:51804)
0923 20:14:53.830110 (+ 340us) negotiation.cc:107] Waiting for socket to connect
0923 20:14:53.830200 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.170.189:34745: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112}
I20260923 20:14:53.831440 24624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:14:53.831060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.189:34745 (local address 127.0.0.1:51814)
0923 20:14:53.831224 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0923 20:14:53.831308 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.170.189:34745: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260923 20:14:53.831709 24621 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.170.189:34745: Network error: Client connection negotiation failed: client connection to 127.1.170.189:34745: connect: Connection refused (error 111)
I20260923 20:14:53.923022 24621 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.170.189" port: 34745 } attempt: 3
I20260923 20:14:53.924280 24624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:14:53.923814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.189:34745 (local address 127.0.0.1:51824)
0923 20:14:53.924041 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0923 20:14:53.924141 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.170.189:34745: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
I20260923 20:14:53.925500 24624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:14:53.925065 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.189:34745 (local address 127.0.0.1:51830)
0923 20:14:53.925278 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0923 20:14:53.925351 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.170.189:34745: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
W20260923 20:14:53.925777 24621 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.170.189:34745: Network error: Client connection negotiation failed: client connection to 127.1.170.189:34745: connect: Connection refused (error 111)
W20260923 20:14:54.068192 24628 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260923 20:14:54.068531 24628 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:14:54.068614 24628 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:14:54.068696 24628 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260923 20:14:54.069018 24621 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.170.189" port: 34745 } attempt: 4
I20260923 20:14:54.070220 24624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:14:54.069665 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.189:34745 (local address 127.0.0.1:51832)
0923 20:14:54.069986 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0923 20:14:54.070084 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.170.189:34745: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":152}
I20260923 20:14:54.071298 24624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:14:54.070942 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.189:34745 (local address 127.0.0.1:51840)
0923 20:14:54.071093 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0923 20:14:54.071178 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.170.189:34745: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260923 20:14:54.071514 24621 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.170.189:34745: Network error: Client connection negotiation failed: client connection to 127.1.170.189:34745: connect: Connection refused (error 111)
W20260923 20:14:54.076614 24628 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 20:14:54.076745 24628 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:14:54.076869 24628 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:14:54.076954 24628 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 20:14:54.077065 24628 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 20:14:54.088119 24628 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.170.148:34397
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790194121577027-1706-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790194121577027-1706-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.1.170.190:33757,127.1.170.189:34745,127.1.170.188:45387
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790194121577027-1706-0/krb5kdc/kudu_127.1.170.189.keytab
--principal=kudu/127.1.170.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.170.189:34745
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790194121577027-1706-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.1.170.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790194121577027-1706-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:14:54.089498 24628 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:14:54.090904 24628 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:14:54.103350 24633 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:14:54.103415 24634 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:14:54.103590 24636 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:14:54.104521 24628 server_base.cc:1034] running on GCE node
I20260923 20:14:54.105345 24628 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:14:54.106887 24628 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:14:54.108103 24628 hybrid_clock.cc:648] HybridClock initialized: now 1790194494108044 us; error 36 us; skew 500 ppm
I20260923 20:14:54.108439 24628 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:14:54 dist-test-slave-th4g krb5kdc[24541](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194494, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.170.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:14:54.111609 24628 init.cc:377] Logged in from keytab as kudu/127.1.170.189@KRBTEST.COM (short username kudu)
I20260923 20:14:54.113089 24628 webserver.cc:460] Webserver started at http://127.1.170.189:32909/ using document root <none> and password file <none>
I20260923 20:14:54.113721 24628 fs_manager.cc:359] Metadata directory not provided
I20260923 20:14:54.113808 24628 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:14:54.114131 24628 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:14:54.117118 24628 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790194121577027-1706-0/minicluster-data/master-1/data/instance:
uuid: "96ac9f6cc3194f3789e9303045580a6f"
format_stamp: "Formatted at 2026-09-23 20:14:54 on dist-test-slave-th4g"
server_key: "9390d28607a26f493806c9e22cf3f3cb"
server_key_iv: "eac8972cdf9950932bf80a8e6d19ad86"
server_key_version: "encryptionkey@0"
I20260923 20:14:54.117842 24628 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790194121577027-1706-0/minicluster-data/master-1/wal/instance:
uuid: "96ac9f6cc3194f3789e9303045580a6f"
format_stamp: "Formatted at 2026-09-23 20:14:54 on dist-test-slave-th4g"
server_key: "9390d28607a26f493806c9e22cf3f3cb"
server_key_iv: "eac8972cdf9950932bf80a8e6d19ad86"
server_key_version: "encryptionkey@0"
I20260923 20:14:54.123273 24628 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.003s
I20260923 20:14:54.126683 24643 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:14:54.127941 24628 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260923 20:14:54.128125 24628 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790194121577027-1706-0/minicluster-data/master-1/data,/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790194121577027-1706-0/minicluster-data/master-1/wal
uuid: "96ac9f6cc3194f3789e9303045580a6f"
format_stamp: "Formatted at 2026-09-23 20:14:54 on dist-test-slave-th4g"
server_key: "9390d28607a26f493806c9e22cf3f3cb"
server_key_iv: "eac8972cdf9950932bf80a8e6d19ad86"
server_key_version: "encryptionkey@0"
I20260923 20:14:54.128333 24628 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790194121577027-1706-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790194121577027-1706-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790194121577027-1706-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:14:54.149171 24628 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:14:54.153158 24628 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:14:54.153432 24628 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:14:54.172430 24628 rpc_server.cc:307] RPC server started. Bound to: 127.1.170.189:34745
I20260923 20:14:54.172437 24694 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.170.189:34745 every 8 connection(s)
I20260923 20:14:54.173794 24628 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790194121577027-1706-0/minicluster-data/master-1/data/info.pb
I20260923 20:14:54.175916 1706 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqUuku6/build/asan/bin/kudu as pid 24628
I20260923 20:14:54.176101 1706 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790194121577027-1706-0/minicluster-data/master-1/wal/instance
I20260923 20:14:54.176581 1706 external_mini_cluster.cc:1428] Setting key b9baf8ac2d884563122ce3c806d9d9e1
I20260923 20:14:54.177626 24695 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:14:54.184230 24695 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.170.190" port: 33757 } has no permanent_uuid. Determining permanent_uuid...
Sep 23 20:14:54 dist-test-slave-th4g krb5kdc[24541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194494, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.170.189@KRBTEST.COM for kudu/127.1.170.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.1.170.188@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.170.188@KRBTEST.COM" created.
I20260923 20:14:54.204953 24625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:14:54.187061 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58120 (local address 127.1.170.190:33757)
0923 20:14:54.187301 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0923 20:14:54.187309 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 20:14:54.188460 (+ 1151us) server_negotiation.cc:407] Connection header received
0923 20:14:54.190380 (+ 1920us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:14:54.190387 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:14:54.190480 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:14:54.190589 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:14:54.192019 (+ 1430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:14:54.192711 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:14:54.194363 (+ 1652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:14:54.194789 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:14:54.194924 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:14:54.198647 (+ 3723us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:14:54.198657 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:14:54.198663 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:14:54.198710 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:14:54.201821 (+ 3111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:14:54.202675 (+ 854us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:14:54.202681 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:14:54.202685 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:14:54.202831 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:14:54.203244 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:14:54.203249 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:14:54.203253 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:14:54.203543 (+ 290us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:14:54.203699 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0923 20:14:54.204410 (+ 711us) server_negotiation.cc:299] Negotiation successful
0923 20:14:54.204633 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99}
I20260923 20:14:54.205988 24698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:14:54.187571 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.190:33757 (local address 127.0.0.1:58120)
0923 20:14:54.188307 (+ 736us) negotiation.cc:107] Waiting for socket to connect
0923 20:14:54.188343 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0923 20:14:54.190118 (+ 1775us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:14:54.190818 (+ 700us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:14:54.190826 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:14:54.191334 (+ 508us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:14:54.191858 (+ 524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:14:54.191874 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:14:54.192879 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:14:54.192883 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:14:54.194202 (+ 1319us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:14:54.194214 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:14:54.195012 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:14:54.195018 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:14:54.195254 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:14:54.195926 (+ 672us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:14:54.195949 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:14:54.198395 (+ 2446us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:14:54.202021 (+ 3626us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:14:54.202028 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:14:54.202041 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:14:54.202547 (+ 506us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:14:54.202944 (+ 397us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:14:54.202948 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:14:54.202950 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:14:54.203116 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:14:54.203684 (+ 568us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:14:54.203692 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:14:54.204034 (+ 342us) client_negotiation.cc:769] Sending connection context
0923 20:14:54.204290 (+ 256us) client_negotiation.cc:240] Negotiation successful
0923 20:14:54.204811 (+ 521us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":511,"thread_start_us":311,"threads_started":1}
I20260923 20:14:54.207743 24695 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.170.189" port: 34745 } has no permanent_uuid. Determining permanent_uuid...
Sep 23 20:14:54 dist-test-slave-th4g krb5kdc[24541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194494, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.170.189@KRBTEST.COM for kudu/127.1.170.189@KRBTEST.COM
I20260923 20:14:54.221906 24698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:14:54.208502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.189:34745 (local address 127.0.0.1:51844)
0923 20:14:54.208729 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0923 20:14:54.208750 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0923 20:14:54.208864 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:14:54.209783 (+ 919us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:14:54.209790 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:14:54.210354 (+ 564us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:14:54.210557 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:14:54.210572 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:14:54.211921 (+ 1349us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:14:54.211928 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:14:54.212760 (+ 832us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:14:54.212771 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:14:54.213582 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:14:54.213587 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:14:54.213756 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:14:54.214394 (+ 638us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:14:54.214417 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:14:54.216306 (+ 1889us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:14:54.219358 (+ 3052us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:14:54.219363 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:14:54.219367 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:14:54.219708 (+ 341us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:14:54.220083 (+ 375us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:14:54.220087 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:14:54.220091 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:14:54.220248 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:14:54.220891 (+ 643us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:14:54.220896 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:14:54.221023 (+ 127us) client_negotiation.cc:769] Sending connection context
0923 20:14:54.221172 (+ 149us) client_negotiation.cc:240] Negotiation successful
0923 20:14:54.221444 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":62}
I20260923 20:14:54.221947 24699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:14:54.208707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51844 (local address 127.1.170.189:34745)
0923 20:14:54.209371 (+ 664us) server_negotiation.cc:206] Beginning negotiation
0923 20:14:54.209377 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:14:54.209401 (+ 24us) server_negotiation.cc:407] Connection header received
0923 20:14:54.209445 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:14:54.209451 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:14:54.209614 (+ 163us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:14:54.209730 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:14:54.210729 (+ 999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:14:54.211789 (+ 1060us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:14:54.212901 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:14:54.213426 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:14:54.213566 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:14:54.216508 (+ 2942us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:14:54.216519 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:14:54.216531 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:14:54.216556 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:14:54.219212 (+ 2656us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:14:54.219861 (+ 649us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:14:54.219868 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:14:54.219870 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:14:54.219969 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:14:54.220352 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:14:54.220359 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:14:54.220363 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:14:54.220747 (+ 384us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:14:54.220956 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0923 20:14:54.221351 (+ 395us) server_negotiation.cc:299] Negotiation successful
0923 20:14:54.221556 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":514,"thread_start_us":373,"threads_started":1}
I20260923 20:14:54.224596 24695 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.170.188" port: 45387 } has no permanent_uuid. Determining permanent_uuid...
I20260923 20:14:54.225724 24698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:14:54.225288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.188:45387 (local address 127.0.0.1:56078)
0923 20:14:54.225471 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0923 20:14:54.225577 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.170.188:45387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260923 20:14:54.226048 24645 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.170.188:45387: connect: Connection refused (error 111)
I20260923 20:14:54.227628 24698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:14:54.227258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.188:45387 (local address 127.0.0.1:56088)
0923 20:14:54.227425 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0923 20:14:54.227519 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.170.188:45387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
W20260923 20:14:54.227958 24695 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.170.188:45387: Network error: Client connection negotiation failed: client connection to 127.1.170.188:45387: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.170.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790194121577027-1706-0/krb5kdc/kudu_127.1.170.188.keytab.
Entry for principal kudu/127.1.170.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790194121577027-1706-0/krb5kdc/kudu_127.1.170.188.keytab.
I20260923 20:14:54.238463 1706 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqUuku6/build/asan/bin/kudu
/tmp/dist-test-taskqUuku6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790194121577027-1706-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790194121577027-1706-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790194121577027-1706-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790194121577027-1706-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.170.188:45387
--webserver_interface=127.1.170.188
--webserver_port=0
--keytab_file=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790194121577027-1706-0/krb5kdc/kudu_127.1.170.188.keytab
--principal=kudu/127.1.170.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.170.148:34397
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.170.190:33757,127.1.170.189:34745,127.1.170.188:45387
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790194121577027-1706-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790194121577027-1706-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790194121577027-1706-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260923 20:14:54.277264 24695 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.170.188" port: 45387 } attempt: 1
I20260923 20:14:54.278468 24698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:14:54.278034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.188:45387 (local address 127.0.0.1:56096)
0923 20:14:54.278223 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0923 20:14:54.278326 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.170.188:45387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260923 20:14:54.279397 24698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:14:54.279071 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.188:45387 (local address 127.0.0.1:56104)
0923 20:14:54.279227 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0923 20:14:54.279298 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.170.188:45387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260923 20:14:54.279593 24695 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.170.188:45387: Network error: Client connection negotiation failed: client connection to 127.1.170.188:45387: connect: Connection refused (error 111)
I20260923 20:14:54.347836 24695 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.170.188" port: 45387 } attempt: 2
I20260923 20:14:54.348974 24698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:14:54.348538 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.188:45387 (local address 127.0.0.1:56120)
0923 20:14:54.348739 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0923 20:14:54.348824 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.170.188:45387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260923 20:14:54.350001 24698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:14:54.349616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.188:45387 (local address 127.0.0.1:56126)
0923 20:14:54.349753 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0923 20:14:54.349828 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.170.188:45387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260923 20:14:54.350206 24695 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.170.188:45387: Network error: Client connection negotiation failed: client connection to 127.1.170.188:45387: connect: Connection refused (error 111)
I20260923 20:14:54.370785 24621 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.170.189" port: 34745 } attempt: 5
Sep 23 20:14:54 dist-test-slave-th4g krb5kdc[24541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194493, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.170.190@KRBTEST.COM for kudu/127.1.170.189@KRBTEST.COM
I20260923 20:14:54.386241 24699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:14:54.371712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51846 (local address 127.1.170.189:34745)
0923 20:14:54.371900 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0923 20:14:54.371905 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 20:14:54.371951 (+ 46us) server_negotiation.cc:407] Connection header received
0923 20:14:54.372210 (+ 259us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:14:54.372214 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:14:54.372301 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:14:54.372405 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:14:54.373463 (+ 1058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:14:54.374469 (+ 1006us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:14:54.376156 (+ 1687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:14:54.376627 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:14:54.376729 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:14:54.380719 (+ 3990us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:14:54.380723 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:14:54.380727 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:14:54.380762 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:14:54.383548 (+ 2786us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:14:54.384321 (+ 773us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:14:54.384325 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:14:54.384327 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:14:54.384439 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:14:54.384828 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:14:54.384831 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:14:54.384833 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:14:54.385099 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:14:54.385208 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0923 20:14:54.385680 (+ 472us) server_negotiation.cc:299] Negotiation successful
0923 20:14:54.385971 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260923 20:14:54.386240 24624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:14:54.371583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.189:34745 (local address 127.0.0.1:51846)
0923 20:14:54.371872 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0923 20:14:54.371892 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0923 20:14:54.372025 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:14:54.372476 (+ 451us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:14:54.372483 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:14:54.373027 (+ 544us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:14:54.373269 (+ 242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:14:54.373282 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:14:54.374617 (+ 1335us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:14:54.374623 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:14:54.375983 (+ 1360us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:14:54.375998 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:14:54.376819 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:14:54.376828 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:14:54.377131 (+ 303us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:14:54.377895 (+ 764us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:14:54.377925 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:14:54.380462 (+ 2537us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:14:54.383689 (+ 3227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:14:54.383695 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:14:54.383699 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:14:54.384207 (+ 508us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:14:54.384556 (+ 349us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:14:54.384561 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:14:54.384565 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:14:54.384726 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:14:54.385214 (+ 488us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:14:54.385219 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:14:54.385396 (+ 177us) client_negotiation.cc:769] Sending connection context
0923 20:14:54.385550 (+ 154us) client_negotiation.cc:240] Negotiation successful
0923 20:14:54.385837 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":164}
I20260923 20:14:54.387498 24621 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.170.188" port: 45387 } has no permanent_uuid. Determining permanent_uuid...
I20260923 20:14:54.388538 24624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:14:54.388129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.188:45387 (local address 127.0.0.1:56130)
0923 20:14:54.388287 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0923 20:14:54.388366 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.170.188:45387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260923 20:14:54.389467 24624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:14:54.389150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.188:45387 (local address 127.0.0.1:56134)
0923 20:14:54.389292 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0923 20:14:54.389374 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.170.188:45387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260923 20:14:54.389679 24621 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.170.188:45387: Network error: Client connection negotiation failed: client connection to 127.1.170.188:45387: connect: Connection refused (error 111)
I20260923 20:14:54.440901 24621 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.170.188" port: 45387 } attempt: 1
I20260923 20:14:54.441427 24695 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.170.188" port: 45387 } attempt: 3
I20260923 20:14:54.442278 24624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:14:54.441678 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.188:45387 (local address 127.0.0.1:56150)
0923 20:14:54.441940 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0923 20:14:54.442048 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.170.188:45387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260923 20:14:54.442754 24698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:14:54.442046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.188:45387 (local address 127.0.0.1:56164)
0923 20:14:54.442252 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0923 20:14:54.442361 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.170.188:45387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260923 20:14:54.443356 24624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:14:54.442956 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.188:45387 (local address 127.0.0.1:56168)
0923 20:14:54.443144 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0923 20:14:54.443227 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.170.188:45387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260923 20:14:54.443784 24621 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.170.188:45387: Network error: Client connection negotiation failed: client connection to 127.1.170.188:45387: connect: Connection refused (error 111)
I20260923 20:14:54.443933 24698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:14:54.443458 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.188:45387 (local address 127.0.0.1:56182)
0923 20:14:54.443710 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0923 20:14:54.443802 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.170.188:45387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":146}
W20260923 20:14:54.444147 24695 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.170.188:45387: Network error: Client connection negotiation failed: client connection to 127.1.170.188:45387: connect: Connection refused (error 111)
I20260923 20:14:54.512032 24621 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.170.188" port: 45387 } attempt: 2
I20260923 20:14:54.513396 24624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:14:54.512770 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.188:45387 (local address 127.0.0.1:56190)
0923 20:14:54.513027 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0923 20:14:54.513179 (+ 152us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.170.188:45387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
I20260923 20:14:54.514607 24624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:14:54.514176 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.188:45387 (local address 127.0.0.1:56206)
0923 20:14:54.514363 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0923 20:14:54.514464 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.170.188:45387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260923 20:14:54.514870 24621 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.170.188:45387: Network error: Client connection negotiation failed: client connection to 127.1.170.188:45387: connect: Connection refused (error 111)
W20260923 20:14:54.572894 24702 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260923 20:14:54.573232 24702 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:14:54.573341 24702 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:14:54.573483 24702 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:14:54.580760 24702 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 20:14:54.580854 24702 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:14:54.580945 24702 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:14:54.581020 24702 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 20:14:54.581131 24702 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 20:14:54.587366 24695 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.170.188" port: 45387 } attempt: 4
I20260923 20:14:54.588683 24698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:14:54.588106 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.188:45387 (local address 127.0.0.1:56208)
0923 20:14:54.588377 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0923 20:14:54.588497 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.170.188:45387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260923 20:14:54.589848 24698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:14:54.589498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.188:45387 (local address 127.0.0.1:56220)
0923 20:14:54.589643 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0923 20:14:54.589723 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.170.188:45387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260923 20:14:54.590108 24695 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.170.188:45387: Network error: Client connection negotiation failed: client connection to 127.1.170.188:45387: connect: Connection refused (error 111)
I20260923 20:14:54.592514 24702 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.170.148:34397
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790194121577027-1706-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790194121577027-1706-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.1.170.190:33757,127.1.170.189:34745,127.1.170.188:45387
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790194121577027-1706-0/krb5kdc/kudu_127.1.170.188.keytab
--principal=kudu/127.1.170.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.170.188:45387
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790194121577027-1706-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.1.170.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790194121577027-1706-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:14:54.593945 24702 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:14:54.595413 24702 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:14:54.607163 24707 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:14:54.607194 24708 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:14:54.607450 24710 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:14:54.608095 24702 server_base.cc:1034] running on GCE node
I20260923 20:14:54.608928 24702 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:14:54.610357 24702 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:14:54.611603 24702 hybrid_clock.cc:648] HybridClock initialized: now 1790194494611530 us; error 45 us; skew 500 ppm
I20260923 20:14:54.611961 24702 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:14:54 dist-test-slave-th4g krb5kdc[24541](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194494, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.170.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:14:54.615185 24702 init.cc:377] Logged in from keytab as kudu/127.1.170.188@KRBTEST.COM (short username kudu)
I20260923 20:14:54.616678 24702 webserver.cc:460] Webserver started at http://127.1.170.188:43713/ using document root <none> and password file <none>
I20260923 20:14:54.617357 24702 fs_manager.cc:359] Metadata directory not provided
I20260923 20:14:54.617486 24702 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:14:54.617918 24702 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:14:54.620931 24702 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790194121577027-1706-0/minicluster-data/master-2/data/instance:
uuid: "db1026647eb149979eb2b0051291227c"
format_stamp: "Formatted at 2026-09-23 20:14:54 on dist-test-slave-th4g"
server_key: "73b7bec8e172972f79221c26d9f63dc6"
server_key_iv: "00503fcbc7c752e8dca14fc5b08b9b2e"
server_key_version: "encryptionkey@0"
I20260923 20:14:54.621130 24621 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.170.188" port: 45387 } attempt: 3
I20260923 20:14:54.621826 24702 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790194121577027-1706-0/minicluster-data/master-2/wal/instance:
uuid: "db1026647eb149979eb2b0051291227c"
format_stamp: "Formatted at 2026-09-23 20:14:54 on dist-test-slave-th4g"
server_key: "73b7bec8e172972f79221c26d9f63dc6"
server_key_iv: "00503fcbc7c752e8dca14fc5b08b9b2e"
server_key_version: "encryptionkey@0"
I20260923 20:14:54.622316 24624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:14:54.621716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.188:45387 (local address 127.0.0.1:56222)
0923 20:14:54.621982 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0923 20:14:54.622095 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.170.188:45387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260923 20:14:54.623440 24624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:14:54.623054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.188:45387 (local address 127.0.0.1:56224)
0923 20:14:54.623255 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0923 20:14:54.623328 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.170.188:45387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97}
W20260923 20:14:54.623687 24621 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.170.188:45387: Network error: Client connection negotiation failed: client connection to 127.1.170.188:45387: connect: Connection refused (error 111)
I20260923 20:14:54.627655 24702 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260923 20:14:54.631217 24717 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:14:54.632809 24702 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260923 20:14:54.632994 24702 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790194121577027-1706-0/minicluster-data/master-2/data,/tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790194121577027-1706-0/minicluster-data/master-2/wal
uuid: "db1026647eb149979eb2b0051291227c"
format_stamp: "Formatted at 2026-09-23 20:14:54 on dist-test-slave-th4g"
server_key: "73b7bec8e172972f79221c26d9f63dc6"
server_key_iv: "00503fcbc7c752e8dca14fc5b08b9b2e"
server_key_version: "encryptionkey@0"
I20260923 20:14:54.633270 24702 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790194121577027-1706-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790194121577027-1706-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790194121577027-1706-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:14:54.648854 24702 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:14:54.653170 24702 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:14:54.653448 24702 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:14:54.672354 24702 rpc_server.cc:307] RPC server started. Bound to: 127.1.170.188:45387
I20260923 20:14:54.672361 24768 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.170.188:45387 every 8 connection(s)
I20260923 20:14:54.674034 24702 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790194121577027-1706-0/minicluster-data/master-2/data/info.pb
I20260923 20:14:54.677671 24769 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:14:54.683813 1706 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqUuku6/build/asan/bin/kudu as pid 24702
I20260923 20:14:54.683996 1706 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqUuku6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790194121577027-1706-0/minicluster-data/master-2/wal/instance
I20260923 20:14:54.684449 1706 external_mini_cluster.cc:1428] Setting key 599d94e2cb58bd055308360cf3dc17ec
I20260923 20:14:54.683902 24769 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.170.190" port: 33757 } has no permanent_uuid. Determining permanent_uuid...
Sep 23 20:14:54 dist-test-slave-th4g krb5kdc[24541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194494, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.170.188@KRBTEST.COM for kudu/127.1.170.190@KRBTEST.COM
Sep 23 20:14:54 dist-test-slave-th4g krb5kdc[24541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194493, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.170.188@KRBTEST.COM
I20260923 20:14:54.704227 24625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:14:54.686918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58124 (local address 127.1.170.190:33757)
0923 20:14:54.687126 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0923 20:14:54.687132 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:14:54.687921 (+ 789us) server_negotiation.cc:407] Connection header received
0923 20:14:54.689571 (+ 1650us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:14:54.689576 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:14:54.689643 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:14:54.689852 (+ 209us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:14:54.691167 (+ 1315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:14:54.691939 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:14:54.693649 (+ 1710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:14:54.694270 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:14:54.694365 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:14:54.698543 (+ 4178us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:14:54.698548 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:14:54.698551 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:14:54.698584 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:14:54.701120 (+ 2536us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:14:54.702001 (+ 881us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:14:54.702007 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:14:54.702011 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:14:54.702148 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:14:54.702572 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:14:54.702581 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:14:54.702583 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:14:54.702836 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:14:54.703005 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0923 20:14:54.703650 (+ 645us) server_negotiation.cc:299] Negotiation successful
0923 20:14:54.703865 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260923 20:14:54.705302 24773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:14:54.686994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.190:33757 (local address 127.0.0.1:58124)
0923 20:14:54.687752 (+ 758us) negotiation.cc:107] Waiting for socket to connect
0923 20:14:54.687796 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0923 20:14:54.689296 (+ 1500us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:14:54.689950 (+ 654us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:14:54.689957 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:14:54.690633 (+ 676us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:14:54.690985 (+ 352us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:14:54.691002 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:14:54.692104 (+ 1102us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:14:54.692110 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:14:54.693504 (+ 1394us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:14:54.693515 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:14:54.694511 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:14:54.694519 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:14:54.694787 (+ 268us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:14:54.695747 (+ 960us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:14:54.695771 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:14:54.698350 (+ 2579us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:14:54.701285 (+ 2935us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:14:54.701297 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:14:54.701309 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:14:54.701840 (+ 531us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:14:54.702273 (+ 433us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:14:54.702278 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:14:54.702282 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:14:54.702452 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:14:54.702976 (+ 524us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:14:54.702984 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:14:54.703297 (+ 313us) client_negotiation.cc:769] Sending connection context
0923 20:14:54.703552 (+ 255us) client_negotiation.cc:240] Negotiation successful
0923 20:14:54.704059 (+ 507us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":523,"mutex_wait_us":41,"thread_start_us":355,"threads_started":1}
I20260923 20:14:54.705320 24772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:14:54.686301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56240 (local address 127.1.170.188:45387)
0923 20:14:54.687351 (+ 1050us) server_negotiation.cc:206] Beginning negotiation
0923 20:14:54.687368 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0923 20:14:54.687423 (+ 55us) server_negotiation.cc:407] Connection header received
0923 20:14:54.689296 (+ 1873us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:14:54.689323 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:14:54.689618 (+ 295us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:14:54.689764 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:14:54.691242 (+ 1478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:14:54.692663 (+ 1421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:14:54.694071 (+ 1408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:14:54.694734 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:14:54.694848 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:14:54.699120 (+ 4272us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:14:54.699132 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:14:54.699149 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:14:54.699189 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:14:54.702188 (+ 2999us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:14:54.702865 (+ 677us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:14:54.702871 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:14:54.702873 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:14:54.702994 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:14:54.703381 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:14:54.703390 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:14:54.703392 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:14:54.703706 (+ 314us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:14:54.704027 (+ 321us) server_negotiation.cc:1035] Waiting for connection context
0923 20:14:54.704360 (+ 333us) server_negotiation.cc:299] Negotiation successful
0923 20:14:54.704598 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":78,"server-negotiator.queue_time_us":782,"thread_start_us":541,"threads_started":1}
I20260923 20:14:54.706959 24769 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.170.189" port: 34745 } has no permanent_uuid. Determining permanent_uuid...
Sep 23 20:14:54 dist-test-slave-th4g krb5kdc[24541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194494, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.170.188@KRBTEST.COM for kudu/127.1.170.189@KRBTEST.COM
Sep 23 20:14:54 dist-test-slave-th4g krb5kdc[24541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194493, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.170.190@KRBTEST.COM
I20260923 20:14:54.721403 24699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:14:54.707769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51850 (local address 127.1.170.189:34745)
0923 20:14:54.708091 (+ 322us) server_negotiation.cc:206] Beginning negotiation
0923 20:14:54.708097 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:14:54.708117 (+ 20us) server_negotiation.cc:407] Connection header received
0923 20:14:54.708197 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:14:54.708201 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:14:54.708268 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:14:54.708384 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:14:54.709265 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:14:54.709994 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:14:54.711276 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:14:54.711692 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:14:54.711796 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:14:54.715158 (+ 3362us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:14:54.715164 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:14:54.715175 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:14:54.715208 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:14:54.718488 (+ 3280us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:14:54.719412 (+ 924us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:14:54.719420 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:14:54.719425 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:14:54.719587 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:14:54.719989 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:14:54.719994 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:14:54.720001 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:14:54.720345 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:14:54.720533 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0923 20:14:54.720950 (+ 417us) server_negotiation.cc:299] Negotiation successful
0923 20:14:54.721131 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":200}
I20260923 20:14:54.721581 24773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:14:54.707811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.189:34745 (local address 127.0.0.1:51850)
0923 20:14:54.707967 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0923 20:14:54.707987 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0923 20:14:54.708086 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:14:54.708437 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:14:54.708441 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:14:54.708916 (+ 475us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:14:54.709116 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:14:54.709129 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:14:54.710129 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:14:54.710137 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:14:54.711132 (+ 995us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:14:54.711148 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:14:54.711845 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:14:54.711852 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:14:54.712070 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:14:54.712681 (+ 611us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:14:54.712698 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:14:54.714968 (+ 2270us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:14:54.718657 (+ 3689us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:14:54.718661 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:14:54.718664 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:14:54.719291 (+ 627us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:14:54.719725 (+ 434us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:14:54.719729 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:14:54.719732 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:14:54.719871 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:14:54.720564 (+ 693us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:14:54.720568 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:14:54.720704 (+ 136us) client_negotiation.cc:769] Sending connection context
0923 20:14:54.720823 (+ 119us) client_negotiation.cc:240] Negotiation successful
0923 20:14:54.721116 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260923 20:14:54.722697 24769 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.170.188" port: 45387 } has no permanent_uuid. Determining permanent_uuid...
I20260923 20:14:54.723809 24625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:14:54.708503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58140 (local address 127.1.170.190:33757)
0923 20:14:54.708662 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0923 20:14:54.708667 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 20:14:54.708720 (+ 53us) server_negotiation.cc:407] Connection header received
0923 20:14:54.708882 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:14:54.708885 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:14:54.708939 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:14:54.709011 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:14:54.710196 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:14:54.710864 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:14:54.712356 (+ 1492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:14:54.712882 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:14:54.712975 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:14:54.717182 (+ 4207us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:14:54.717192 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:14:54.717198 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:14:54.717248 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:14:54.720830 (+ 3582us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:14:54.721861 (+ 1031us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:14:54.721866 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:14:54.721869 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:14:54.722029 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:14:54.722403 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:14:54.722415 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:14:54.722419 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:14:54.722771 (+ 352us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:14:54.723053 (+ 282us) server_negotiation.cc:1035] Waiting for connection context
0923 20:14:54.723366 (+ 313us) server_negotiation.cc:299] Negotiation successful
0923 20:14:54.723530 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
Sep 23 20:14:54 dist-test-slave-th4g krb5kdc[24541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194494, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.170.188@KRBTEST.COM for kudu/127.1.170.188@KRBTEST.COM
Sep 23 20:14:54 dist-test-slave-th4g krb5kdc[24541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194493, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.170.189@KRBTEST.COM
I20260923 20:14:54.737951 24773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:14:54.723433 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.188:45387 (local address 127.0.0.1:56246)
0923 20:14:54.724001 (+ 568us) negotiation.cc:107] Waiting for socket to connect
0923 20:14:54.724016 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0923 20:14:54.724133 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:14:54.724483 (+ 350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:14:54.724486 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:14:54.724823 (+ 337us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:14:54.724935 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:14:54.724942 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:14:54.726257 (+ 1315us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:14:54.726261 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:14:54.727236 (+ 975us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:14:54.727251 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:14:54.727934 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:14:54.727942 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:14:54.728148 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:14:54.728756 (+ 608us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:14:54.728774 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:14:54.731345 (+ 2571us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:14:54.734949 (+ 3604us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:14:54.734954 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:14:54.734959 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:14:54.735466 (+ 507us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:14:54.735922 (+ 456us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:14:54.735928 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:14:54.735931 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:14:54.736147 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:14:54.736829 (+ 682us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:14:54.736834 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:14:54.737009 (+ 175us) client_negotiation.cc:769] Sending connection context
0923 20:14:54.737170 (+ 161us) client_negotiation.cc:240] Negotiation successful
0923 20:14:54.737499 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":472}
I20260923 20:14:54.738783 24772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:14:54.723713 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56246 (local address 127.1.170.188:45387)
0923 20:14:54.723885 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0923 20:14:54.723891 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:14:54.724062 (+ 171us) server_negotiation.cc:407] Connection header received
0923 20:14:54.724226 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:14:54.724243 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:14:54.724311 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:14:54.724358 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:14:54.725073 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:14:54.726128 (+ 1055us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:14:54.727376 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:14:54.727782 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:14:54.727880 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:14:54.731546 (+ 3666us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:14:54.731552 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:14:54.731557 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:14:54.731596 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:14:54.734827 (+ 3231us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:14:54.735595 (+ 768us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:14:54.735602 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:14:54.735606 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:14:54.735805 (+ 199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:14:54.736329 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:14:54.736333 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:14:54.736335 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:14:54.736684 (+ 349us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:14:54.736880 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0923 20:14:54.738262 (+ 1382us) server_negotiation.cc:299] Negotiation successful
0923 20:14:54.738477 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260923 20:14:54.741315 24699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:14:54.725375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51854 (local address 127.1.170.189:34745)
0923 20:14:54.725544 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0923 20:14:54.725548 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 20:14:54.725564 (+ 16us) server_negotiation.cc:407] Connection header received
0923 20:14:54.725606 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:14:54.725609 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:14:54.725666 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:14:54.725739 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:14:54.727070 (+ 1331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:14:54.727909 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:14:54.729079 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:14:54.729708 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:14:54.729859 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:14:54.734931 (+ 5072us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:14:54.734937 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:14:54.734941 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:14:54.734981 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:14:54.738503 (+ 3522us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:14:54.739325 (+ 822us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:14:54.739329 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:14:54.739331 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:14:54.739456 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:14:54.739865 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:14:54.739870 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:14:54.739873 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:14:54.740209 (+ 336us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:14:54.740479 (+ 270us) server_negotiation.cc:1035] Waiting for connection context
0923 20:14:54.740745 (+ 266us) server_negotiation.cc:299] Negotiation successful
0923 20:14:54.740982 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260923 20:14:54.744619 24769 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P db1026647eb149979eb2b0051291227c: Bootstrap starting.
I20260923 20:14:54.746680 1706 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 20:14:54.748528 24769 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P db1026647eb149979eb2b0051291227c: Neither blocks nor log segments found. Creating new log.
I20260923 20:14:54.749832 24769 log.cc:824] T 00000000000000000000000000000000 P db1026647eb149979eb2b0051291227c: Log is configured to *not* fsync() on all Append() calls
I20260923 20:14:54.752763 24769 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P db1026647eb149979eb2b0051291227c: No bootstrap required, opened a new log
I20260923 20:14:54.749816 24596 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "3d45e0d3fb0447afbff64afec99359d1"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58140
I20260923 20:14:54.759922 24769 raft_consensus.cc:348] T 00000000000000000000000000000000 P db1026647eb149979eb2b0051291227c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d45e0d3fb0447afbff64afec99359d1" member_type: VOTER last_known_addr { host: "127.1.170.190" port: 33757 } } peers { permanent_uuid: "96ac9f6cc3194f3789e9303045580a6f" member_type: VOTER last_known_addr { host: "127.1.170.189" port: 34745 } } peers { permanent_uuid: "db1026647eb149979eb2b0051291227c" member_type: VOTER last_known_addr { host: "127.1.170.188" port: 45387 } }
I20260923 20:14:54.760257 24769 raft_consensus.cc:729] T 00000000000000000000000000000000 P db1026647eb149979eb2b0051291227c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db1026647eb149979eb2b0051291227c, State: Initialized, Role: FOLLOWER
I20260923 20:14:54.760675 24769 consensus_queue.cc:260] T 00000000000000000000000000000000 P db1026647eb149979eb2b0051291227c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d45e0d3fb0447afbff64afec99359d1" member_type: VOTER last_known_addr { host: "127.1.170.190" port: 33757 } } peers { permanent_uuid: "96ac9f6cc3194f3789e9303045580a6f" member_type: VOTER last_known_addr { host: "127.1.170.189" port: 34745 } } peers { permanent_uuid: "db1026647eb149979eb2b0051291227c" member_type: VOTER last_known_addr { host: "127.1.170.188" port: 45387 } }
I20260923 20:14:54.761525 24775 sys_catalog.cc:455] T 00000000000000000000000000000000 P db1026647eb149979eb2b0051291227c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d45e0d3fb0447afbff64afec99359d1" member_type: VOTER last_known_addr { host: "127.1.170.190" port: 33757 } } peers { permanent_uuid: "96ac9f6cc3194f3789e9303045580a6f" member_type: VOTER last_known_addr { host: "127.1.170.189" port: 34745 } } peers { permanent_uuid: "db1026647eb149979eb2b0051291227c" member_type: VOTER last_known_addr { host: "127.1.170.188" port: 45387 } } }
I20260923 20:14:54.761850 24775 sys_catalog.cc:458] T 00000000000000000000000000000000 P db1026647eb149979eb2b0051291227c [sys.catalog]: This master's current role is: FOLLOWER
I20260923 20:14:54.762833 24769 sys_catalog.cc:564] T 00000000000000000000000000000000 P db1026647eb149979eb2b0051291227c [sys.catalog]: configured and running, proceeding with master startup.
W20260923 20:14:54.772498 24786 catalog_manager.cc:1558] T 00000000000000000000000000000000 P db1026647eb149979eb2b0051291227c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260923 20:14:54.772652 24786 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260923 20:14:54.770386 24702 master_runner.cc:186] Error getting master registration for 127.1.170.190:33757: OK, instance_id { permanent_uuid: "3d45e0d3fb0447afbff64afec99359d1" instance_seqno: 1790194493658719 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260923 20:14:54.781694 24702 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260923 20:14:54.800911 24621 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.170.188" port: 45387 } attempt: 4
Sep 23 20:14:54 dist-test-slave-th4g krb5kdc[24541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194493, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.170.190@KRBTEST.COM for kudu/127.1.170.188@KRBTEST.COM
I20260923 20:14:54.818284 24624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:14:54.801698 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.188:45387 (local address 127.0.0.1:56260)
0923 20:14:54.802026 (+ 328us) negotiation.cc:107] Waiting for socket to connect
0923 20:14:54.802047 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0923 20:14:54.802170 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:14:54.802576 (+ 406us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:14:54.802583 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:14:54.803081 (+ 498us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:14:54.803324 (+ 243us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:14:54.803336 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:14:54.804775 (+ 1439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:14:54.804780 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:14:54.806427 (+ 1647us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:14:54.806447 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:14:54.807488 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:14:54.807495 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:14:54.807724 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:14:54.808583 (+ 859us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:14:54.808614 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:14:54.811395 (+ 2781us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:14:54.815563 (+ 4168us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:14:54.815569 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:14:54.815574 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:14:54.816111 (+ 537us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:14:54.816545 (+ 434us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:14:54.816550 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:14:54.816554 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:14:54.816715 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:14:54.817461 (+ 746us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:14:54.817465 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:14:54.817603 (+ 138us) client_negotiation.cc:769] Sending connection context
0923 20:14:54.817712 (+ 109us) client_negotiation.cc:240] Negotiation successful
0923 20:14:54.818003 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":190}
I20260923 20:14:54.818483 24772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:14:54.801814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56260 (local address 127.1.170.188:45387)
0923 20:14:54.802096 (+ 282us) server_negotiation.cc:206] Beginning negotiation
0923 20:14:54.802102 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:14:54.802126 (+ 24us) server_negotiation.cc:407] Connection header received
0923 20:14:54.802321 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:14:54.802326 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:14:54.802410 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:14:54.802533 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:14:54.803483 (+ 950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:14:54.804616 (+ 1133us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:14:54.806613 (+ 1997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:14:54.807325 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:14:54.807468 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:14:54.811601 (+ 4133us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:14:54.811609 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:14:54.811615 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:14:54.811656 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:14:54.815376 (+ 3720us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:14:54.816272 (+ 896us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:14:54.816278 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:14:54.816282 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:14:54.816423 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:14:54.816866 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:14:54.816882 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:14:54.816885 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:14:54.817349 (+ 464us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:14:54.817526 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0923 20:14:54.817860 (+ 334us) server_negotiation.cc:299] Negotiation successful
0923 20:14:54.818109 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120}
I20260923 20:14:54.824620 24621 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3d45e0d3fb0447afbff64afec99359d1: Bootstrap starting.
I20260923 20:14:54.828739 24621 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3d45e0d3fb0447afbff64afec99359d1: Neither blocks nor log segments found. Creating new log.
I20260923 20:14:54.830067 24621 log.cc:824] T 00000000000000000000000000000000 P 3d45e0d3fb0447afbff64afec99359d1: Log is configured to *not* fsync() on all Append() calls
I20260923 20:14:54.833489 24621 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3d45e0d3fb0447afbff64afec99359d1: No bootstrap required, opened a new log
I20260923 20:14:54.841429 24621 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3d45e0d3fb0447afbff64afec99359d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d45e0d3fb0447afbff64afec99359d1" member_type: VOTER last_known_addr { host: "127.1.170.190" port: 33757 } } peers { permanent_uuid: "96ac9f6cc3194f3789e9303045580a6f" member_type: VOTER last_known_addr { host: "127.1.170.189" port: 34745 } } peers { permanent_uuid: "db1026647eb149979eb2b0051291227c" member_type: VOTER last_known_addr { host: "127.1.170.188" port: 45387 } }
I20260923 20:14:54.841908 24621 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3d45e0d3fb0447afbff64afec99359d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d45e0d3fb0447afbff64afec99359d1, State: Initialized, Role: FOLLOWER
I20260923 20:14:54.842412 24621 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3d45e0d3fb0447afbff64afec99359d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d45e0d3fb0447afbff64afec99359d1" member_type: VOTER last_known_addr { host: "127.1.170.190" port: 33757 } } peers { permanent_uuid: "96ac9f6cc3194f3789e9303045580a6f" member_type: VOTER last_known_addr { host: "127.1.170.189" port: 34745 } } peers { permanent_uuid: "db1026647eb149979eb2b0051291227c" member_type: VOTER last_known_addr { host: "127.1.170.188" port: 45387 } }
I20260923 20:14:54.843474 24789 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3d45e0d3fb0447afbff64afec99359d1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d45e0d3fb0447afbff64afec99359d1" member_type: VOTER last_known_addr { host: "127.1.170.190" port: 33757 } } peers { permanent_uuid: "96ac9f6cc3194f3789e9303045580a6f" member_type: VOTER last_known_addr { host: "127.1.170.189" port: 34745 } } peers { permanent_uuid: "db1026647eb149979eb2b0051291227c" member_type: VOTER last_known_addr { host: "127.1.170.188" port: 45387 } } }
I20260923 20:14:54.843859 24789 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3d45e0d3fb0447afbff64afec99359d1 [sys.catalog]: This master's current role is: FOLLOWER
I20260923 20:14:54.844673 24621 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3d45e0d3fb0447afbff64afec99359d1 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 20:14:54.844758 24596 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3d45e0d3fb0447afbff64afec99359d1 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260923 20:14:54.844990 24596 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3d45e0d3fb0447afbff64afec99359d1 [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:14:54.847901 24596 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3d45e0d3fb0447afbff64afec99359d1 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d45e0d3fb0447afbff64afec99359d1" member_type: VOTER last_known_addr { host: "127.1.170.190" port: 33757 } } peers { permanent_uuid: "96ac9f6cc3194f3789e9303045580a6f" member_type: VOTER last_known_addr { host: "127.1.170.189" port: 34745 } } peers { permanent_uuid: "db1026647eb149979eb2b0051291227c" member_type: VOTER last_known_addr { host: "127.1.170.188" port: 45387 } }
I20260923 20:14:54.850222 24744 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3d45e0d3fb0447afbff64afec99359d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "db1026647eb149979eb2b0051291227c"
I20260923 20:14:54.850792 24744 raft_consensus.cc:3037] T 00000000000000000000000000000000 P db1026647eb149979eb2b0051291227c [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:14:54.854409 24744 raft_consensus.cc:2454] T 00000000000000000000000000000000 P db1026647eb149979eb2b0051291227c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d45e0d3fb0447afbff64afec99359d1 in term 1.
I20260923 20:14:54.854843 24596 leader_election.cc:288] T 00000000000000000000000000000000 P 3d45e0d3fb0447afbff64afec99359d1 [CANDIDATE]: Term 1 election: Requested vote from peers 96ac9f6cc3194f3789e9303045580a6f (127.1.170.189:34745), db1026647eb149979eb2b0051291227c (127.1.170.188:45387)
I20260923 20:14:54.855206 24571 leader_election.cc:302] T 00000000000000000000000000000000 P 3d45e0d3fb0447afbff64afec99359d1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d45e0d3fb0447afbff64afec99359d1, db1026647eb149979eb2b0051291227c; no voters:
I20260923 20:14:54.855990 24789 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3d45e0d3fb0447afbff64afec99359d1 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 20:14:54.856693 24789 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3d45e0d3fb0447afbff64afec99359d1 [term 1 LEADER]: Becoming Leader. State: Replica: 3d45e0d3fb0447afbff64afec99359d1, State: Running, Role: LEADER
I20260923 20:14:54.857303 24789 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3d45e0d3fb0447afbff64afec99359d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d45e0d3fb0447afbff64afec99359d1" member_type: VOTER last_known_addr { host: "127.1.170.190" port: 33757 } } peers { permanent_uuid: "96ac9f6cc3194f3789e9303045580a6f" member_type: VOTER last_known_addr { host: "127.1.170.189" port: 34745 } } peers { permanent_uuid: "db1026647eb149979eb2b0051291227c" member_type: VOTER last_known_addr { host: "127.1.170.188" port: 45387 } }
I20260923 20:14:54.861310 24798 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3d45e0d3fb0447afbff64afec99359d1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3d45e0d3fb0447afbff64afec99359d1. Latest consensus state: current_term: 1 leader_uuid: "3d45e0d3fb0447afbff64afec99359d1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d45e0d3fb0447afbff64afec99359d1" member_type: VOTER last_known_addr { host: "127.1.170.190" port: 33757 } } peers { permanent_uuid: "96ac9f6cc3194f3789e9303045580a6f" member_type: VOTER last_known_addr { host: "127.1.170.189" port: 34745 } } peers { permanent_uuid: "db1026647eb149979eb2b0051291227c" member_type: VOTER last_known_addr { host: "127.1.170.188" port: 45387 } } }
I20260923 20:14:54.862033 24798 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3d45e0d3fb0447afbff64afec99359d1 [sys.catalog]: This master's current role is: LEADER
I20260923 20:14:54.879380 24803 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 20:14:54.889451 24695 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.170.188" port: 45387 } attempt: 5
W20260923 20:14:54.898365 24575 tablet.cc:2369] T 00000000000000000000000000000000 P 3d45e0d3fb0447afbff64afec99359d1: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 23 20:14:54 dist-test-slave-th4g krb5kdc[24541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194494, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.170.189@KRBTEST.COM for kudu/127.1.170.188@KRBTEST.COM
W20260923 20:14:54.905572 24723 tablet.cc:2369] T 00000000000000000000000000000000 P db1026647eb149979eb2b0051291227c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 20:14:54.919349 24625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:14:54.887039 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58154 (local address 127.1.170.190:33757)
0923 20:14:54.887277 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0923 20:14:54.887287 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0923 20:14:54.887810 (+ 523us) server_negotiation.cc:407] Connection header received
0923 20:14:54.888322 (+ 512us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:14:54.888329 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:14:54.888418 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:14:54.888498 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:14:54.893635 (+ 5137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:14:54.894789 (+ 1154us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:14:54.900246 (+ 5457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:14:54.900974 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:14:54.901080 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:14:54.910183 (+ 9103us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:14:54.910191 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:14:54.910196 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:14:54.910244 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:14:54.913935 (+ 3691us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:14:54.915819 (+ 1884us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:14:54.915825 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:14:54.915829 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:14:54.916018 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:14:54.916850 (+ 832us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:14:54.916856 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:14:54.916860 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:14:54.917244 (+ 384us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:14:54.917473 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0923 20:14:54.918714 (+ 1241us) server_negotiation.cc:299] Negotiation successful
0923 20:14:54.918981 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260923 20:14:54.922037 24772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:14:54.891121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56274 (local address 127.1.170.188:45387)
0923 20:14:54.891658 (+ 537us) server_negotiation.cc:206] Beginning negotiation
0923 20:14:54.891667 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 20:14:54.891695 (+ 28us) server_negotiation.cc:407] Connection header received
0923 20:14:54.891771 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:14:54.891779 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:14:54.891892 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:14:54.892017 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:14:54.893456 (+ 1439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:14:54.895157 (+ 1701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:14:54.897485 (+ 2328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:14:54.898413 (+ 928us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:14:54.898858 (+ 445us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:14:54.908015 (+ 9157us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:14:54.908026 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:14:54.908034 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:14:54.908093 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:14:54.913355 (+ 5262us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:14:54.915799 (+ 2444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:14:54.915809 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:14:54.915815 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:14:54.916034 (+ 219us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:14:54.917409 (+ 1375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:14:54.917417 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:14:54.917421 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:14:54.917779 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:14:54.918046 (+ 267us) server_negotiation.cc:1035] Waiting for connection context
0923 20:14:54.919834 (+ 1788us) server_negotiation.cc:299] Negotiation successful
0923 20:14:54.920137 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":388}
I20260923 20:14:54.923998 24698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:14:54.890160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.170.188:45387 (local address 127.0.0.1:56274)
0923 20:14:54.890697 (+ 537us) negotiation.cc:107] Waiting for socket to connect
0923 20:14:54.890723 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0923 20:14:54.890860 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:14:54.892274 (+ 1414us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:14:54.892282 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:14:54.892948 (+ 666us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:14:54.893205 (+ 257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:14:54.893221 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:14:54.895367 (+ 2146us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:14:54.895375 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:14:54.897265 (+ 1890us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:14:54.897286 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:14:54.899375 (+ 2089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:14:54.899384 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:14:54.899684 (+ 300us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:14:54.900877 (+ 1193us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:14:54.900911 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:14:54.906707 (+ 5796us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:14:54.914714 (+ 8007us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:14:54.914722 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:14:54.914729 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:14:54.915514 (+ 785us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:14:54.917077 (+ 1563us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:14:54.917082 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:14:54.917085 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:14:54.917267 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:14:54.918421 (+ 1154us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:14:54.918429 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:14:54.918607 (+ 178us) client_negotiation.cc:769] Sending connection context
0923 20:14:54.918797 (+ 190us) client_negotiation.cc:240] Negotiation successful
0923 20:14:54.919167 (+ 370us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":378}
I20260923 20:14:54.931969 24695 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 96ac9f6cc3194f3789e9303045580a6f: Bootstrap starting.
I20260923 20:14:54.934190 24699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:14:54.888392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51866 (local address 127.1.170.189:34745)
0923 20:14:54.888630 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0923 20:14:54.888639 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 20:14:54.909138 (+ 20499us) server_negotiation.cc:407] Connection header received
0923 20:14:54.909435 (+ 297us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:14:54.909442 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:14:54.909532 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:14:54.909611 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:14:54.911542 (+ 1931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:14:54.912723 (+ 1181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:14:54.916230 (+ 3507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:14:54.917125 (+ 895us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:14:54.917223 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:14:54.925390 (+ 8167us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:14:54.925398 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:14:54.925402 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:14:54.925444 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:14:54.929211 (+ 3767us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:14:54.930590 (+ 1379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:14:54.930595 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:14:54.930598 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:14:54.930773 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:14:54.931573 (+ 800us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:14:54.931578 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:14:54.931581 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:14:54.931990 (+ 409us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:14:54.932331 (+ 341us) server_negotiation.cc:1035] Waiting for connection context
0923 20:14:54.933417 (+ 1086us) server_negotiation.cc:299] Negotiation successful
0923 20:14:54.933692 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260923 20:14:54.936833 24695 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 96ac9f6cc3194f3789e9303045580a6f: Neither blocks nor log segments found. Creating new log.
I20260923 20:14:54.945438 24695 log.cc:824] T 00000000000000000000000000000000 P 96ac9f6cc3194f3789e9303045580a6f: Log is configured to *not* fsync() on all Append() calls
I20260923 20:14:54.958977 24803 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 20:14:54.970248 24813 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260923 20:14:54.973551 24695 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 96ac9f6cc3194f3789e9303045580a6f: No bootstrap required, opened a new log
I20260923 20:14:54.984033 24772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:14:54.891312 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56286 (local address 127.1.170.188:45387)
0923 20:14:54.922968 (+ 31656us) server_negotiation.cc:206] Beginning negotiation
0923 20:14:54.922974 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:14:54.922995 (+ 21us) server_negotiation.cc:407] Connection header received
0923 20:14:54.923057 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:14:54.923063 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:14:54.923172 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:14:54.923292 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:14:54.938514 (+ 15222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:14:54.940039 (+ 1525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:14:54.944536 (+ 4497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:14:54.946602 (+ 2066us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:14:54.946787 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:14:54.961248 (+ 14461us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:14:54.961261 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:14:54.961269 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:14:54.961348 (+ 79us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:14:54.966063 (+ 4715us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:14:54.970074 (+ 4011us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:14:54.970084 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:14:54.970090 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:14:54.970260 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:14:54.978005 (+ 7745us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:14:54.978014 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:14:54.978020 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:14:54.978488 (+ 468us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:14:54.978779 (+ 291us) server_negotiation.cc:1035] Waiting for connection context
0923 20:14:54.983279 (+ 4500us) server_negotiation.cc:299] Negotiation successful
0923 20:14:54.983579 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31533,"thread_start_us":52607,"threads_started":1}
I20260923 20:14:54.974224 24554 master_runner.cc:186] Error getting master registration for 127.1.170.189:34745: OK, instance_id { permanent_uuid: "96ac9f6cc3194f3789e9303045580a6f" instance_seqno: 1790194494167127 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260923 20:14:54.985307 24554 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260923 20:14:54.994176 24695 raft_consensus.cc:348] T 00000000000000000000000000000000 P 96ac9f6cc3194f3789e9303045580a6f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d45e0d3fb0447afbff64afec99359d1" member_type: VOTER last_known_addr { host: "127.1.170.190" port: 33757 } } peers { permanent_uuid: "96ac9f6cc3194f3789e9303045580a6f" member_type: VOTER last_known_addr { host: "127.1.170.189" port: 34745 } } peers { permanent_uuid: "db1026647eb149979eb2b0051291227c" member_type: VOTER last_known_addr { host: "127.1.170.188" port: 45387 } }
I20260923 20:14:54.994571 24744 raft_consensus.cc:1261] T 00000000000000000000000000000000 P db1026647eb149979eb2b0051291227c [term 1 FOLLOWER]: Refusing update from remote peer 3d45e0d3fb0447afbff64afec99359d1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 20:14:54.994598 24695 raft_consensus.cc:729] T 00000000000000000000000000000000 P 96ac9f6cc3194f3789e9303045580a6f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 96ac9f6cc3194f3789e9303045580a6f, State: Initialized, Role: FOLLOWER
I20260923 20:14:54.995123 24695 consensus_queue.cc:260] T 00000000000000000000000000000000 P 96ac9f6cc3194f3789e9303045580a6f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d45e0d3fb0447afbff64afec99359d1" member_type: VOTER last_known_addr { host: "127.1.170.190" port: 33757 } } peers { permanent_uuid: "96ac9f6cc3194f3789e9303045580a6f" member_type: VOTER last_known_addr { host: "127.1.170.189" port: 34745 } } peers { permanent_uuid: "db1026647eb149979eb2b0051291227c" member_type: VOTER last_known_addr { host: "127.1.170.188" port: 45387 } }
I20260923 20:14:54.995571 24798 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 3d45e0d3fb0447afbff64afec99359d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db1026647eb149979eb2b0051291227c" member_type: VOTER last_known_addr { host: "127.1.170.188" port: 45387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 20:14:54.996407 24815 sys_catalog.cc:455] T 00000000000000000000000000000000 P 96ac9f6cc3194f3789e9303045580a6f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d45e0d3fb0447afbff64afec99359d1" member_type: VOTER last_known_addr { host: "127.1.170.190" port: 33757 } } peers { permanent_uuid: "96ac9f6cc3194f3789e9303045580a6f" member_type: VOTER last_known_addr { host: "127.1.170.189" port: 34745 } } peers { permanent_uuid: "db1026647eb149979eb2b0051291227c" member_type: VOTER last_known_addr { host: "127.1.170.188" port: 45387 } } }
I20260923 20:14:54.997098 24815 sys_catalog.cc:458] T 00000000000000000000000000000000 P 96ac9f6cc3194f3789e9303045580a6f [sys.catalog]: This master's current role is: FOLLOWER
I20260923 20:14:55.012053 24670 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3d45e0d3fb0447afbff64afec99359d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "96ac9f6cc3194f3789e9303045580a6f"
I20260923 20:14:55.012488 24669 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 96ac9f6cc3194f3789e9303045580a6f [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:14:55.012013 24695 sys_catalog.cc:564] T 00000000000000000000000000000000 P 96ac9f6cc3194f3789e9303045580a6f [sys.catalog]: configured and running, proceeding with master startup.
I20260923 20:14:55.016645 24669 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 96ac9f6cc3194f3789e9303045580a6f [term 1 FOLLOWER]: Refusing update from remote peer 3d45e0d3fb0447afbff64afec99359d1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 20:14:55.018836 24775 sys_catalog.cc:455] T 00000000000000000000000000000000 P db1026647eb149979eb2b0051291227c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3d45e0d3fb0447afbff64afec99359d1. Latest consensus state: current_term: 1 leader_uuid: "3d45e0d3fb0447afbff64afec99359d1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d45e0d3fb0447afbff64afec99359d1" member_type: VOTER last_known_addr { host: "127.1.170.190" port: 33757 } } peers { permanent_uuid: "96ac9f6cc3194f3789e9303045580a6f" member_type: VOTER last_known_addr { host: "127.1.170.189" port: 34745 } } peers { permanent_uuid: "db1026647eb149979eb2b0051291227c" member_type: VOTER last_known_addr { host: "127.1.170.188" port: 45387 } } }
I20260923 20:14:55.019132 24775 sys_catalog.cc:458] T 00000000000000000000000000000000 P db1026647eb149979eb2b0051291227c [sys.catalog]: This master's current role is: FOLLOWER
I20260923 20:14:55.021320 24814 mvcc.cc:205] Tried to move back new op lower bound from 7332636651478126592 to 7332636650941214720. Current Snapshot: MvccSnapshot[applied={T|T < 7332636651478126592}]
I20260923 20:14:55.023212 24789 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3d45e0d3fb0447afbff64afec99359d1 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "3d45e0d3fb0447afbff64afec99359d1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d45e0d3fb0447afbff64afec99359d1" member_type: VOTER last_known_addr { host: "127.1.170.190" port: 33757 } } peers { permanent_uuid: "96ac9f6cc3194f3789e9303045580a6f" member_type: VOTER last_known_addr { host: "127.1.170.189" port: 34745 } } peers { permanent_uuid: "db1026647eb149979eb2b0051291227c" member_type: VOTER last_known_addr { host: "127.1.170.188" port: 45387 } } }
I20260923 20:14:55.023566 24789 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3d45e0d3fb0447afbff64afec99359d1 [sys.catalog]: This master's current role is: LEADER
I20260923 20:14:55.020242 24670 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 96ac9f6cc3194f3789e9303045580a6f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d45e0d3fb0447afbff64afec99359d1 in term 1.
I20260923 20:14:55.026561 24798 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 3d45e0d3fb0447afbff64afec99359d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "96ac9f6cc3194f3789e9303045580a6f" member_type: VOTER last_known_addr { host: "127.1.170.189" port: 34745 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 20:14:55.027429 24803 catalog_manager.cc:1347] Generated new cluster ID: 7a84e719754141d4b2fbf335e9c176a4
I20260923 20:14:55.027555 24803 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 20:14:55.030118 24775 sys_catalog.cc:455] T 00000000000000000000000000000000 P db1026647eb149979eb2b0051291227c [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "3d45e0d3fb0447afbff64afec99359d1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d45e0d3fb0447afbff64afec99359d1" member_type: VOTER last_known_addr { host: "127.1.170.190" port: 33757 } } peers { permanent_uuid: "96ac9f6cc3194f3789e9303045580a6f" member_type: VOTER last_known_addr { host: "127.1.170.189" port: 34745 } } peers { permanent_uuid: "db1026647eb149979eb2b0051291227c" member_type: VOTER last_known_addr { host: "127.1.170.188" port: 45387 } } }
I20260923 20:14:55.030370 24775 sys_catalog.cc:458] T 00000000000000000000000000000000 P db1026647eb149979eb2b0051291227c [sys.catalog]: This master's current role is: FOLLOWER
I20260923 20:14:55.034420 24816 mvcc.cc:205] Tried to move back new op lower bound from 7332636651478126592 to 7332636650941214720. Current Snapshot: MvccSnapshot[applied={T|T < 7332636651478126592 or (T in {7332636651478126592})}]
I20260923 20:14:55.047379 24803 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 20:14:55.049382 24803 catalog_manager.cc:1504] Loading token signing keys...
I20260923 20:14:55.067075 24803 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3d45e0d3fb0447afbff64afec99359d1: Generated new TSK 0
I20260923 20:14:55.068455 24803 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 20:14:55.103636 24825 sys_catalog.cc:455] T 00000000000000000000000000000000 P 96ac9f6cc3194f3789e9303045580a6f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "3d45e0d3fb0447afbff64afec99359d1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d45e0d3fb0447afbff64afec99359d1" member_type: VOTER last_known_addr { host: "127.1.170.190" port: 33757 } } peers { permanent_uuid: "96ac9f6cc3194f3789e9303045580a6f" member_type: VOTER last_known_addr { host: "127.1.170.189" port: 34745 } } peers { permanent_uuid: "db1026647eb149979eb2b0051291227c" member_type: VOTER last_known_addr { host: "127.1.170.188" port: 45387 } } }
I20260923 20:14:55.104056 24825 sys_catalog.cc:458] T 00000000000000000000000000000000 P 96ac9f6cc3194f3789e9303045580a6f [sys.catalog]: This master's current role is: FOLLOWER
I20260923 20:14:55.104458 24815 sys_catalog.cc:455] T 00000000000000000000000000000000 P 96ac9f6cc3194f3789e9303045580a6f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3d45e0d3fb0447afbff64afec99359d1. Latest consensus state: current_term: 1 leader_uuid: "3d45e0d3fb0447afbff64afec99359d1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d45e0d3fb0447afbff64afec99359d1" member_type: VOTER last_known_addr { host: "127.1.170.190" port: 33757 } } peers { permanent_uuid: "96ac9f6cc3194f3789e9303045580a6f" member_type: VOTER last_known_addr { host: "127.1.170.189" port: 34745 } } peers { permanent_uuid: "db1026647eb149979eb2b0051291227c" member_type: VOTER last_known_addr { host: "127.1.170.188" port: 45387 } } }
I20260923 20:14:55.104700 24815 sys_catalog.cc:458] T 00000000000000000000000000000000 P 96ac9f6cc3194f3789e9303045580a6f [sys.catalog]: This master's current role is: FOLLOWER
I20260923 20:14:55.106201 24798 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3d45e0d3fb0447afbff64afec99359d1 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "3d45e0d3fb0447afbff64afec99359d1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d45e0d3fb0447afbff64afec99359d1" member_type: VOTER last_known_addr { host: "127.1.170.190" port: 33757 } } peers { permanent_uuid: "96ac9f6cc3194f3789e9303045580a6f" member_type: VOTER last_known_addr { host: "127.1.170.189" port: 34745 } } peers { permanent_uuid: "db1026647eb149979eb2b0051291227c" member_type: VOTER last_known_addr { host: "127.1.170.188" port: 45387 } } }
I20260923 20:14:55.106467 24798 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3d45e0d3fb0447afbff64afec99359d1 [sys.catalog]: This master's current role is: LEADER
I20260923 20:14:55.125862 24837 catalog_manager.cc:1259] Loaded cluster ID: 7a84e719754141d4b2fbf335e9c176a4
I20260923 20:14:55.126060 24837 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 96ac9f6cc3194f3789e9303045580a6f: loading cluster ID for follower catalog manager: success
I20260923 20:14:55.129621 24837 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 96ac9f6cc3194f3789e9303045580a6f: acquiring CA information for follower catalog manager: success
I20260923 20:14:55.136459 24837 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 96ac9f6cc3194f3789e9303045580a6f: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260923 20:14:55.212672 24625 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 20:14:55.194423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58156 (local address 127.1.170.190:33757)
0923 20:14:55.194634 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0923 20:14:55.194640 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:14:55.210532 (+ 15892us) server_negotiation.cc:407] Connection header received
0923 20:14:55.210904 (+ 372us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:14:55.210912 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:14:55.211009 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:14:55.211096 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:14:55.212378 (+ 1282us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58156: BlockingRecv error: recv got EOF from 127.0.0.1:58156 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
W20260923 20:14:55.214093 24699 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 20:14:55.200646 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51868 (local address 127.1.170.189:34745)
0923 20:14:55.200867 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0923 20:14:55.200874 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:14:55.210940 (+ 10066us) server_negotiation.cc:407] Connection header received
0923 20:14:55.211844 (+ 904us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:14:55.211851 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:14:55.211947 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:14:55.212019 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:14:55.213856 (+ 1837us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51868: BlockingRecv error: recv got EOF from 127.0.0.1:51868 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
W20260923 20:14:55.215343 24772 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 20:14:55.211640 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56292 (local address 127.1.170.188:45387)
0923 20:14:55.211890 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0923 20:14:55.211896 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:14:55.213742 (+ 1846us) server_negotiation.cc:407] Connection header received
0923 20:14:55.214026 (+ 284us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:14:55.214032 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:14:55.214114 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:14:55.214195 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:14:55.215083 (+ 888us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56292: BlockingRecv error: recv got EOF from 127.0.0.1:56292 (error 108)
Metrics: {"server-negotiator.queue_time_us":132}
I20260923 20:14:55.278357 24699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:14:55.262200 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51874 (local address 127.1.170.189:34745)
0923 20:14:55.262414 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0923 20:14:55.262421 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:14:55.270193 (+ 7772us) server_negotiation.cc:407] Connection header received
0923 20:14:55.270578 (+ 385us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:14:55.270589 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:14:55.270804 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:14:55.270950 (+ 146us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 20:14:55.271710 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:14:55.272928 (+ 1218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:14:55.275207 (+ 2279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:14:55.275973 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:14:55.276153 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:14:55.276772 (+ 619us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 20:14:55.277507 (+ 735us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 20:14:55.277637 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0923 20:14:55.277909 (+ 272us) server_negotiation.cc:299] Negotiation successful
0923 20:14:55.278006 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
W20260923 20:14:55.326679 24772 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 20:14:55.312489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56306 (local address 127.1.170.188:45387)
0923 20:14:55.312730 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0923 20:14:55.312737 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:14:55.324065 (+ 11328us) server_negotiation.cc:407] Connection header received
0923 20:14:55.324913 (+ 848us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:14:55.324920 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:14:55.325059 (+ 139us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:14:55.325179 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:14:55.326418 (+ 1239us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56306: BlockingRecv error: recv got EOF from 127.0.0.1:56306 (error 108)
Metrics: {"server-negotiator.queue_time_us":91}
I20260923 20:14:55.378716 24699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:14:55.363574 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51882 (local address 127.1.170.189:34745)
0923 20:14:55.363811 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0923 20:14:55.363818 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:14:55.371400 (+ 7582us) server_negotiation.cc:407] Connection header received
0923 20:14:55.371946 (+ 546us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:14:55.371954 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:14:55.372093 (+ 139us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:14:55.372162 (+ 69us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 20:14:55.372940 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:14:55.374193 (+ 1253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:14:55.376289 (+ 2096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:14:55.376987 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:14:55.377095 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:14:55.377757 (+ 662us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 20:14:55.377966 (+ 209us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 20:14:55.378031 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0923 20:14:55.378282 (+ 251us) server_negotiation.cc:299] Negotiation successful
0923 20:14:55.378379 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103}
W20260923 20:14:55.420939 24772 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 20:14:55.414309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56312 (local address 127.1.170.188:45387)
0923 20:14:55.414507 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0923 20:14:55.414514 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:14:55.419139 (+ 4625us) server_negotiation.cc:407] Connection header received
0923 20:14:55.419435 (+ 296us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:14:55.419441 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:14:55.419545 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:14:55.419686 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:14:55.420624 (+ 938us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56312: BlockingRecv error: recv got EOF from 127.0.0.1:56312 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
I20260923 20:14:55.435320 24699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:14:55.428018 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51890 (local address 127.1.170.189:34745)
0923 20:14:55.428260 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0923 20:14:55.428270 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0923 20:14:55.428342 (+ 72us) server_negotiation.cc:407] Connection header received
0923 20:14:55.428643 (+ 301us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:14:55.428650 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:14:55.428758 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:14:55.428905 (+ 147us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 20:14:55.429350 (+ 445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:14:55.430639 (+ 1289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:14:55.432689 (+ 2050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:14:55.433527 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:14:55.433686 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:14:55.434184 (+ 498us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 20:14:55.434365 (+ 181us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 20:14:55.434480 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0923 20:14:55.434644 (+ 164us) server_negotiation.cc:299] Negotiation successful
0923 20:14:55.434754 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
W20260923 20:14:55.448133 24772 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 20:14:55.442832 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56328 (local address 127.1.170.188:45387)
0923 20:14:55.443050 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0923 20:14:55.443057 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:14:55.446567 (+ 3510us) server_negotiation.cc:407] Connection header received
0923 20:14:55.446772 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:14:55.446778 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:14:55.446881 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:14:55.446994 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:14:55.447865 (+ 871us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56328: BlockingRecv error: recv got EOF from 127.0.0.1:56328 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
I20260923 20:14:55.462132 24699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:14:55.455331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51894 (local address 127.1.170.189:34745)
0923 20:14:55.455574 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0923 20:14:55.455581 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:14:55.456089 (+ 508us) server_negotiation.cc:407] Connection header received
0923 20:14:55.456289 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:14:55.456295 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:14:55.456385 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:14:55.456501 (+ 116us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 20:14:55.456937 (+ 436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:14:55.458137 (+ 1200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:14:55.460104 (+ 1967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:14:55.460748 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:14:55.460869 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:14:55.461356 (+ 487us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 20:14:55.461483 (+ 127us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 20:14:55.461567 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0923 20:14:55.461733 (+ 166us) server_negotiation.cc:299] Negotiation successful
0923 20:14:55.461816 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122}
W20260923 20:14:55.474613 24772 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 20:14:55.472192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56340 (local address 127.1.170.188:45387)
0923 20:14:55.472487 (+ 295us) server_negotiation.cc:206] Beginning negotiation
0923 20:14:55.472495 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 20:14:55.473040 (+ 545us) server_negotiation.cc:407] Connection header received
0923 20:14:55.473244 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:14:55.473251 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:14:55.473345 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:14:55.473459 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:14:55.474384 (+ 925us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56340: BlockingRecv error: recv got EOF from 127.0.0.1:56340 (error 108)
Metrics: {"server-negotiator.queue_time_us":148}
I20260923 20:14:55.493922 24699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:14:55.486663 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51904 (local address 127.1.170.189:34745)
0923 20:14:55.486896 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0923 20:14:55.486904 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 20:14:55.487483 (+ 579us) server_negotiation.cc:407] Connection header received
0923 20:14:55.487687 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:14:55.487694 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:14:55.487788 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:14:55.487896 (+ 108us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 20:14:55.488388 (+ 492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:14:55.489582 (+ 1194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:14:55.491547 (+ 1965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:14:55.492282 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:14:55.492463 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:14:55.492995 (+ 532us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 20:14:55.493203 (+ 208us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 20:14:55.493312 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0923 20:14:55.493474 (+ 162us) server_negotiation.cc:299] Negotiation successful
0923 20:14:55.493564 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
W20260923 20:14:55.508886 24772 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 20:14:55.505810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56344 (local address 127.1.170.188:45387)
0923 20:14:55.506040 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0923 20:14:55.506046 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:14:55.507177 (+ 1131us) server_negotiation.cc:407] Connection header received
0923 20:14:55.507384 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:14:55.507389 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:14:55.507494 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:14:55.507621 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:14:55.508596 (+ 975us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56344: BlockingRecv error: recv got EOF from 127.0.0.1:56344 (error 108)
Metrics: {"server-negotiator.queue_time_us":104}
I20260923 20:14:55.541569 24699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:14:55.532457 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51914 (local address 127.1.170.189:34745)
0923 20:14:55.532691 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0923 20:14:55.532697 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:14:55.533227 (+ 530us) server_negotiation.cc:407] Connection header received
0923 20:14:55.533596 (+ 369us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:14:55.533603 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:14:55.533701 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:14:55.533776 (+ 75us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 20:14:55.535034 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:14:55.536334 (+ 1300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:14:55.538623 (+ 2289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:14:55.539571 (+ 948us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:14:55.539808 (+ 237us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:14:55.540587 (+ 779us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 20:14:55.540805 (+ 218us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 20:14:55.540941 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0923 20:14:55.541145 (+ 204us) server_negotiation.cc:299] Negotiation successful
0923 20:14:55.541268 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
W20260923 20:14:55.555127 24772 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 20:14:55.552234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56350 (local address 127.1.170.188:45387)
0923 20:14:55.552438 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0923 20:14:55.552447 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 20:14:55.553084 (+ 637us) server_negotiation.cc:407] Connection header received
0923 20:14:55.553385 (+ 301us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:14:55.553390 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:14:55.553494 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:14:55.553635 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:14:55.554860 (+ 1225us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56350: BlockingRecv error: recv got EOF from 127.0.0.1:56350 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
I20260923 20:14:55.603658 24699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:14:55.595627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51916 (local address 127.1.170.189:34745)
0923 20:14:55.595894 (+ 267us) server_negotiation.cc:206] Beginning negotiation
0923 20:14:55.595901 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:14:55.596527 (+ 626us) server_negotiation.cc:407] Connection header received
0923 20:14:55.596772 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:14:55.596778 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:14:55.596932 (+ 154us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:14:55.597051 (+ 119us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 20:14:55.597603 (+ 552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:14:55.598889 (+ 1286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:14:55.600935 (+ 2046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:14:55.601677 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:14:55.601853 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:14:55.602404 (+ 551us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 20:14:55.602567 (+ 163us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 20:14:55.602676 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0923 20:14:55.602892 (+ 216us) server_negotiation.cc:299] Negotiation successful
0923 20:14:55.602986 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114}
W20260923 20:14:55.617715 24772 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 20:14:55.615291 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56354 (local address 127.1.170.188:45387)
0923 20:14:55.615487 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0923 20:14:55.615493 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:14:55.616053 (+ 560us) server_negotiation.cc:407] Connection header received
0923 20:14:55.616273 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:14:55.616279 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:14:55.616399 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:14:55.616523 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:14:55.617466 (+ 943us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56354: BlockingRecv error: recv got EOF from 127.0.0.1:56354 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
I20260923 20:14:55.691195 24699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:14:55.685571 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51928 (local address 127.1.170.189:34745)
0923 20:14:55.686249 (+ 678us) server_negotiation.cc:206] Beginning negotiation
0923 20:14:55.686255 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:14:55.686494 (+ 239us) server_negotiation.cc:407] Connection header received
0923 20:14:55.686787 (+ 293us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:14:55.686793 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:14:55.686871 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:14:55.686978 (+ 107us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 20:14:55.687424 (+ 446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:14:55.688129 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:14:55.689474 (+ 1345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:14:55.689952 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:14:55.690094 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:14:55.690490 (+ 396us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 20:14:55.690592 (+ 102us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 20:14:55.690691 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0923 20:14:55.690901 (+ 210us) server_negotiation.cc:299] Negotiation successful
0923 20:14:55.690983 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":481}
W20260923 20:14:55.700435 24772 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 20:14:55.698301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56358 (local address 127.1.170.188:45387)
0923 20:14:55.698582 (+ 281us) server_negotiation.cc:206] Beginning negotiation
0923 20:14:55.698588 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:14:55.698859 (+ 271us) server_negotiation.cc:407] Connection header received
0923 20:14:55.699103 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:14:55.699108 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:14:55.699193 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:14:55.699349 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:14:55.700194 (+ 845us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56358: BlockingRecv error: recv got EOF from 127.0.0.1:56358 (error 108)
Metrics: {"server-negotiator.queue_time_us":112}
I20260923 20:14:55.774936 24786 catalog_manager.cc:1259] Loaded cluster ID: 7a84e719754141d4b2fbf335e9c176a4
I20260923 20:14:55.775043 24786 catalog_manager.cc:1552] T 00000000000000000000000000000000 P db1026647eb149979eb2b0051291227c: loading cluster ID for follower catalog manager: success
I20260923 20:14:55.777577 24786 catalog_manager.cc:1574] T 00000000000000000000000000000000 P db1026647eb149979eb2b0051291227c: acquiring CA information for follower catalog manager: success
I20260923 20:14:55.779560 24786 catalog_manager.cc:1602] T 00000000000000000000000000000000 P db1026647eb149979eb2b0051291227c: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260923 20:14:55.840073 24699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:14:55.833646 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51932 (local address 127.1.170.189:34745)
0923 20:14:55.833936 (+ 290us) server_negotiation.cc:206] Beginning negotiation
0923 20:14:55.833943 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:14:55.834317 (+ 374us) server_negotiation.cc:407] Connection header received
0923 20:14:55.834563 (+ 246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:14:55.834569 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:14:55.834689 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:14:55.834798 (+ 109us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 20:14:55.835383 (+ 585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:14:55.836451 (+ 1068us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:14:55.838325 (+ 1874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:14:55.838756 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:14:55.838900 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:14:55.839329 (+ 429us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 20:14:55.839446 (+ 117us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 20:14:55.839555 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0923 20:14:55.839707 (+ 152us) server_negotiation.cc:299] Negotiation successful
0923 20:14:55.839799 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260923 20:14:55.853165 24772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:14:55.847227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56370 (local address 127.1.170.188:45387)
0923 20:14:55.847459 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0923 20:14:55.847466 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:14:55.847684 (+ 218us) server_negotiation.cc:407] Connection header received
0923 20:14:55.847828 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:14:55.847832 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:14:55.848011 (+ 179us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:14:55.848116 (+ 105us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 20:14:55.848572 (+ 456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:14:55.849329 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:14:55.850834 (+ 1505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:14:55.851273 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:14:55.851375 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:14:55.851935 (+ 560us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 20:14:55.852472 (+ 537us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 20:14:55.852612 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0923 20:14:55.852736 (+ 124us) server_negotiation.cc:299] Negotiation successful
0923 20:14:55.852808 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260923 20:14:55.868060 24942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:14:55.859988 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58166 (local address 127.1.170.190:33757)
0923 20:14:55.861318 (+ 1330us) server_negotiation.cc:206] Beginning negotiation
0923 20:14:55.861323 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 20:14:55.861340 (+ 17us) server_negotiation.cc:407] Connection header received
0923 20:14:55.861439 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:14:55.861443 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:14:55.861597 (+ 154us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:14:55.861795 (+ 198us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 20:14:55.862539 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:14:55.863595 (+ 1056us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:14:55.865550 (+ 1955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:14:55.866056 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:14:55.866202 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:14:55.866799 (+ 597us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 20:14:55.867285 (+ 486us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 20:14:55.867391 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0923 20:14:55.867611 (+ 220us) server_negotiation.cc:299] Negotiation successful
0923 20:14:55.867700 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1215,"thread_start_us":1098,"threads_started":1}
I20260923 20:14:55.868759 24772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:14:55.860865 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56378 (local address 127.1.170.188:45387)
0923 20:14:55.861214 (+ 349us) server_negotiation.cc:206] Beginning negotiation
0923 20:14:55.861223 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 20:14:55.861802 (+ 579us) server_negotiation.cc:407] Connection header received
0923 20:14:55.862158 (+ 356us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:14:55.862163 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:14:55.862259 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:14:55.862335 (+ 76us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 20:14:55.863220 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:14:55.864239 (+ 1019us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:14:55.866588 (+ 2349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:14:55.867227 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:14:55.867391 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:14:55.867911 (+ 520us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 20:14:55.868052 (+ 141us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 20:14:55.868128 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0923 20:14:55.868392 (+ 264us) server_negotiation.cc:299] Negotiation successful
0923 20:14:55.868483 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":188}
I20260923 20:14:55.869428 24699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:14:55.860329 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51942 (local address 127.1.170.189:34745)
0923 20:14:55.861020 (+ 691us) server_negotiation.cc:206] Beginning negotiation
0923 20:14:55.861030 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0923 20:14:55.862534 (+ 1504us) server_negotiation.cc:407] Connection header received
0923 20:14:55.862791 (+ 257us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:14:55.862796 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:14:55.862863 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:14:55.862976 (+ 113us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 20:14:55.863612 (+ 636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:14:55.864778 (+ 1166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:14:55.866969 (+ 2191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:14:55.867505 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:14:55.867626 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:14:55.868485 (+ 859us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 20:14:55.868612 (+ 127us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 20:14:55.868698 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0923 20:14:55.869106 (+ 408us) server_negotiation.cc:299] Negotiation successful
0923 20:14:55.869192 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260923 20:14:55.874956 1706 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskqUuku6/build/asan/bin/kudu pid 24554
I20260923 20:14:55.876137 24616 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:14:55.990128 1706 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqUuku6/build/asan/bin/kudu with pid 24554
I20260923 20:14:56.013098 1706 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskqUuku6/build/asan/bin/kudu pid 24628
I20260923 20:14:56.014742 24690 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:14:56.129093 1706 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqUuku6/build/asan/bin/kudu with pid 24628
I20260923 20:14:56.150081 1706 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskqUuku6/build/asan/bin/kudu pid 24702
I20260923 20:14:56.151279 24764 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:14:56.265249 1706 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqUuku6/build/asan/bin/kudu with pid 24702
2026-09-23T20:14:56Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (4445 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (8930 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (374628 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST